# data[].attributes.attachment_link.readable_text
A comprehensive guide to the basics of Python, a high-level programming language with a simple philosophy

First of all, what is Python? According to its creator, Guido van Rossum, Python is:

"a high-level programming language, and its core design philosophy is all about code readability and a syntax which allows programmers to express concepts in a few lines of code."

For me, the first reason to learn Python was that it is, in fact, a beautifulprogramming language. It was really natural to code in it and express my thoughts.

Another reason was that we can use coding in Python in multiple ways: data science, web development, and machine learning all shine here. Quora, Pinterest, and Spotify all use Python for their backend web development. So let's learn a bit about it.

The Basics

1. Variables

You can think about variables as words that store a value. Simple as that.

In Python, it is really easy to define a variable and set a value to it. Imagine you want to store number 1 in a variable called "one." Let's do it:

How simple was that? You just assigned the value 1 to the variable "one."

And you can assign any other value to whatever other variables you want. As you see in the table above, the variable " two" stores the integer 2, and " some_number" stores 10,000.

Besides integers, we can also use booleans (True / False), strings, float, and so many other data types.

2. Control flow: Conditional statements

" If " uses an expression to evaluate whether a statement is True or False. If it is True, it executes what is inside the "if" statement. For example:

You can also use an " elif " statement:

3. Looping/Iterator

In Python, we can iterate in different forms. I'll talk about two: whileand for.

The while loop needs a " loop condition." If it stays True, it continues iterating. In this example, when num is 11 the loop condition equals False.

Another basic bit of code to better understand is:

The loop condition is True so it keeps iterating - until we set it to False.

For Looping: you apply the variable " num" to the block, and the " for" statement will iterate it for you. This code will print the same as while code: from 1 to 10.

See? It's so simple. The range starts with 1 and goes until the 11th element ( 10 is the 10 th element).

List: Collection | Array | Data Structure

Imagine you want to store the integer 1 in a variable. But maybe now you want to store 2. And 3, 4, 5, and so on.

Do I have another way to store all the integers that I want, but not in millions of variables? You guessed it - there is indeed another way to store them.

List is a collection that can be used to store a list of values (like these integers that you want). So let's use it:

It is really simple. We created an array and stored it on my_integer.

But maybe you are asking: "How can I get a value from this array?"

Great question. List has a concept called index. The first element gets index 0 (zero). The second gets 1, and so on. You get the idea.

To make it clearer, we can represent the array and each element with its index. I can draw it:

Using the Python syntax, it's also simple to understand:

Imagine that you don't want to store integers. You just want to store strings, like a list of your relatives' names. Mine would look something like this:

It works the same way as integers. Nice.

We just learned how Lists indices work. But I still need to show you how we can add an element to the List data structure (an item to a list).

The most common method to add a new value to a List is append. Let's see how it works:

append is super simple. You just need to apply the element (eg. " The Effective Engineer") as the append parameter.

Well, enough about Lists. Let's talk about another data structure.

Dictionary: Key-Value Data Structure

Now we know that Lists are indexed with integer numbers. But what if we don't want to use integer numbers as indices? Some data structures that we can use are numeric, string, or other types of indices.

Let's learn about the Dictionary data structure. Dictionary is a collection of key-value pairs. Here's what it looks like:

The key is the index pointing to the value. How do we access the Dictionary value? You guessed it - using the key. Let's try it:

I created a Dictionary about me. My name, nickname, and nationality. Those attributes are the Dictionary keys.

As we learned how to access the List using an index, we also use indices ( keys in the Dictionary context) to access the value stored in the Dictionary.

In the example, I printed a phrase about me using all the values stored in the Dictionary. Pretty simple, right?

Another cool thing about Dictionary is that we can use anything like the value. In the DictionaryI created, I want to add the key "age" and my real integer age in it:

Here we have a key (age) value (24) pair using string as the key and integer as the value.

As we did with Lists, let's learn how to add elements to a Dictionary. The keypointing to a value is a big part of what Dictionary is. This is also true when we are talking about adding elements to it:

We just need to assign a value to a Dictionary key. Nothing complicated here, right?

Iteration: Looping Through Data Structures

As we learned in the Python Basics, the List iteration is very simple. We Pythondevelopers commonly use For looping. Let's do it:

So for each book in the bookshelf, we can print it. Pretty simple and intuitive. That's Python.

For a hash data structure, we can also use the for loop, but we apply the key :

This is an example of how to use it. For each key in the dictionary , we print the key and it's corresponding value.

Another way to do it is to use the iteritems method.

We did name the two parameters as key and value, but it is not necessary. We can name them anything. Let's see it:

We can see we used the attribute as a parameter for the Dictionary key, and it works properly. Great!

Classes & Objects

A little bit of theory:

Cars have data, like the number of wheels, number of doors, and seating capacity. They also exhibit behavior: they can accelerate, stop, show how much fuel is left, and so many other things.

We identify data as attributes and behavior as methods in object-oriented programming. Again:

Data → Attributes and Behavior → Methods

And a Class is the blueprint from which individual objects are created. In the real world, we often find many objects with the same type. Like cars. All the same make and model (and all have an engine, wheels, doors, and so on). Each car was built from the same set of blueprints and has the same components.

Python Object-Oriented programming mode: ON

Python, as an Object-Oriented programming language, has these concepts: class and object.

A class is a blueprint, a model for its objects.

So again, a class is just a model or a way to define attributes and behavior (as we talked about in the theory section). As an example, a vehicle class has its own attributes that define what objects are vehicles. The number of wheels, type of tank, seating capacity, and maximum velocity is all attributes of a vehicle.

With this in mind, let's look at Python syntax for classes:

We define classes with a class statement -  and that's it. Easy, isn't it?

Objects are instances of a class. We create an instance by naming the class.

Here car is an object (or instance) of the class Vehicle.

Remember that our vehicle class has four attributes: the number of wheels, type of tank, seating capacity, and maximum velocity. We set all these attributes when creating a vehicle object. So here, we define our class to receive data when it initiates it:

We use the init method. We call it a constructor method. So when we create the vehicle object, we can define these attributes. Imagine that we love the Tesla Model S, and we want to create this kind of object. It has four wheels, runs on electric energy, has space for five seats, and the maximum velocity is 250km/hour (155 mph). Let's create this object:

Four wheels + electric "tank type" + five seats + 250km/hour maximum speed.

This is an implementation of two methods: number_of_wheels and set_number_of_wheels. We call it getter & setter. Because the first gets the attribute value, and the second sets a new value for the attribute.

In Python, we can do that using @property ( decorators) to define getters and setters. Let's see it with code:

And we can use these methods as attributes:

This is slightly different than defining methods. The methods work as attributes. For example, when we set the new number of wheels, we don't apply two as a parameter but set the value 2 to number_of_wheels. This is one way to write pythonic getter and setter code.

But we can also use methods for other things, like the " make_noise " method. Let's see it:

When we call this method, it just returns a string "VRRRRUUUUM."

Encapsulation: Hiding Information

Encapsulation is a mechanism that restricts direct access to objects' data and methods. But at the same time, it facilitates operation on that data (objects' methods).

"Encapsulation can be used to hide data members and members function. Under this definition, encapsulation means that the internal representation of an object is generally hidden from view outside of the object's definition." - Wikipedia

All internal representation of an object is hidden from the outside. Only the object can interact with its internal data.

First, we need to understand how public and non-public instance variables and methods work.

Public instance variables

For a Python class, we can initialize a public instance variable within our constructor method. Let's see this:

Within the constructor method:

Here we apply the first_name value as an argument to the public instance variable.

Within the class:

Here, we do not need to apply the first_name as an argument, and all instance objects will have a class attribute initialized with TK.

Cool. We have now learned that we can use public instance variables and class attributes. Another interesting thing about the public part is that we can manage the variable value. What do I mean by that? Our object can manage its variable value: Get and Set variable values.

Keeping the Person class in mind, we want to set another value to its first_name variable:

There we go. We just set another value ( kaio) to the first_name instance variable and it updated the value. Simple as that. Since it's a public variable, we can do that.

Non-public instance variable

We don't use the term "private" here, since no attribute is really private in Python (without a generally unnecessary amount of work). - PEP 8

As the public instance variable , we can define the non-public instance variable both within the constructor method or within the class. The syntax difference is: for non-public instance variables , use an underscore (_) before the variable name.

"'Private' instance variables that cannot be accessed except from inside an object don't exist in Python. However, there is a convention that is followed by most Python code: a name prefixed with an underscore (e.g. _spam) should be treated as a non-public part of the API (whether it is a function, a method or a data member)" - Python Software Foundation

Here's an example:

Did you see the email variable? This is how we define a non-public variable :

We can access and update it. Non-public variables are just a convention and should be treated as a non-public part of the API.

So we use a method that allows us to do it inside our class definition. Let's implement two methods ( email and update_email) to understand it:

Now we can update and access non-public variables using those methods. Let's see:

1. We initiated a new object with first_name TK and email tk@mail.com

2. Printed the email by accessing the non-public variable with a method

3. Tried to set a new email out of our class

4. We need to treat non-public variable as non-public part of the API

5. Updated the non-public variable with our instance method

6. Success! We can update it inside our class with the helper method

Public method

With public methods, we can also use them out of our class:

Let's test it:

Great - we can use it without any problem.

Non-public method

But with non-public methods we aren't able to do it. Let's implement the same Person class, but now with a show_age non-public method using an underscore (_).

And now, we'll try to call this non-public method with our object:

We can access and update it. Non-public methods are just a convention and should be treated as a non-public part of the API.

Here's an example of how we can use it:

Here we have a _get_age non-public method and a show_age public method. The show_age can be used by our object (out of our class) and the _get_age only used inside our class definition (inside show_age method). But again: as a matter of convention.

Encapsulation Summary

With encapsulation, we can ensure that the internal representation of the object is hidden from the outside.

Inheritance: Behaviors and Characteristics

Certain objects have some things in common: their behavior and characteristics.

For example, I inherited some characteristics and behaviors of my father. I inherited his eyes and hair as characteristics and his impatience and introversion as behaviors.

In object-oriented programming, classes can inherit common characteristics (data) and behavior (methods) from another class.

Let's see another example and implement it in Python.

Imagine a car. The number of wheels, seating capacity, and maximum velocity are all attributes of a car. We can say that an ElectricCar class inherits these same attributes from the regular Car class.

Our Car class implemented:

Once initiated, we can use all instance variables created. Nice.

In Python, we apply a parent class to the child class as a parameter. An ElectricCar class can inherit from our Car class.

Simple as that. We don't need to implement any other method, because this class already has it (inherited from Car class). Let's prove it:

Beautiful.

That's It!

We learned a lot of things about Python basics:

How Python variables work

How Python conditional statements work

How Python looping (while & for) works

How to use Lists: Collection | Array

Dictionary Key-Value Collection

How we can iterate through these data structures

Objects and Classes

Attributes as objects' data

Methods as objects' behavior

Using Python getters and setters & property decorator

Encapsulation: hiding information

Inheritance: behaviors and characteristics

Congrats! You completed this dense piece of content about Python.

If you want a complete Python course, learn more real-world coding skills and build projects, try One Month Python Bootcamp. See you there ☺

For more stories and posts about my journey learning & mastering programming, follow my publication The Renaissance Developer.

Have fun, keep learning, and always keep coding.

# data[].attributes.attachment_link.description
A comprehensive guide to the basics of Python, a high-level programming language with a simple philosophy First of all, what is Python? According to its creator, Guido van Rossum, Python is: "a high-level programming language, and its core design philosophy is all about code readability and a syntax which allows programmers to express concepts in a few lines of code."

# data[].attributes.attachment_link.readable_text
A clear and comprehensive blueprint for absolutely anyone who wants to build a simple machine learning model

Even you can build a machine learning model.

Seriously!

Good data alone doesn't always tell the whole story. Are you trying to figure out what someone's salary should be based on their years of experience? Do you need to examine how much you're spending on advertising in relation to your yearly sales? Linear regression might be exactly what you need!

What is linear regression?

Linear regression looks at the relationship between the data you have and the data you want to predict.

Linear Regression is a basic and commonly used type of predictive analysis. It's the most widely used of all statistical techniques. It quantifies the relationship between one or more predictor variables and one outcome variable.

Linear regression models are used to show (or predict) the relationship between two variables or factors. Regression analysis is commonly used to show the correlation between two variables.

You could, for example, look at some information about players on a baseball team and predict how well they might do that season. You might want to examine some variables about a company and predict well their stock might do. You might even just want to examine the number of hours people study and how well they do on a test, or you could look at student's homework grades overall in relation to how well they might do on their tests. It's a seriously useful technique!

Just remember: correlation is not causation! Just because a relationship exists between two variables doesn't mean that one variable caused the other variable! Regression analysis is not used to predict cause-and-effect relationships. It can look at how variables relate to each other. It can examine to what extent variables are associated with each other. It's up to you to take a closer look at those relationships.

A couple of important terms:

The variable that the equation in your linear regression model is predicting is called the dependent variable. We call that one y. The variables that are being used to predict the dependent variable are called the independent variables. We call them X.

You can think of it as though the prediction ( y) is dependent on the other variables ( X). That makes y the dependent variable!

In simple linear regression analysis, each observation consists of two variables. These are the independent variable and the dependent variable. Multiple regression analysis looks at two or more independent variables and how they correlate to the independent variable. The equation that describes how y is related to X is called the regression model!

Regression was first studied in depth by Sir Francis Galton, a man with a wide variety of interests. While he was a very problematic character with a lot of beliefs worth disagreeing with, he did write some books with cool information about things like treating spear wounds and getting your horse unstuck from quicksand. He also did some useful work with fingerprints, hearing tests, and even devised the first weather map. He was knighted in 1909.

While studying data on the relative sizes between parents and their children in plants and animals, he observed that larger-than-average parents have larger-than-average children, but those children will be less large in terms of their relative position within their own generation. He called it regression towards mediocrity. That would be regression to the mean in modern terms.

(I have to say, though, that there is a certain sparkle to the phrase, "regression towards mediocrity" that I need to work into my day-to-day life...)

To be clear, though, we're talking about expectations (predictions) and not absolute certainty!

What good are regression models?

Regression models are used for predicting a real value, for example, salary or height. If your independent variable is time, then you are forecasting future values. Otherwise, your model is predicting present but unknown values. Examples of regression techniques include:

Simple regression

Multiple regression

Polynomial regression

Support Vector Regression

Let's say you're looking at some data that includes employee's years of experience and salary. You want to look at the correlation between those two figures. Maybe you're running a new business or small company that has been kind of setting the numbers randomly.

So how can you find the correlation between those two variables? In order to figure that out, we'll create a model that will tell us what is the best fitting line for this relationship.

Intuition

Here's a simple linear regression formula:

(You might recognize this as the equation for a slope or trend line from high school algebra.)

In this equation, y is the dependent variable, which is what you're trying to explain. For the rest of this article, y will be an employee's salary after a certain number of years of experience.

You can see the independent variable above. That's the variable that is associated with the change in your predicted values. The independent variable might be causing the change or simply associated with the change. Remember, linear regression doesn't prove causation!

The coefficient is how you explain that a change in your independent variable is maybe not totally equal to a change in y.

Now we want to look at the evidence. We want to put a line through our data that best fits our data. A regression line can show a positive linear relationship (the line looks like it's sloping up), a negative linear relationship (the line is sloping down), or really no relationship at all (a flat line).

The constant is the point where the line crosses the vertical axis. For example, if you looked at 0 years of experience in the graph below, your salary would be around $30,000. So the constant in the chart below would be about $30,000.

The steeper the slope, the more money you get for your years of experience. For example, maybe with 1 more year of experience, your salary (y) goes up an additional $10,000, but with a steeper slope, you might wind up with more like $15,000. With a negative slope, you'd actually lose money as you gained experience, but I really hope you won't be working for that company for long...

How does simple linear regression find that line?

When we look at a graph, we can draw vertical lines from the line to our actual observations. You can see the actual observations as the dots, while the line displays the model observations (the predictions).

The line that we drew is the difference between what an employee is actually earning and what he's modeled (predicted) to be earning. We would look at the minimum sum of squares to find the best line, which just means that you'd take the sum of all the squared differences and find the minimum.

That's called the ordinary least squares method!

So how do we do that?

First the imports!

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

Now let's preprocess our data! If you don't know much about data cleaning and preprocessing, you might want to check out this article. It will walk you through importing libraries, preparing your data, and feature scaling.

We're going to copy and paste the code from that article and make two tiny changes. We'll need to change the name of our dataset, of course. Then we'll take a look at the data. For our example, let's say for our employees we have one column of years of experience and one column of salaries and that's it. Keeping in mind that our index starts at 0, we will go ahead and separate the last column from our data for the dependent variable, just like we already have set up. This time, however, we'd be grabbing the second column for our independent variable, so we'd make a minor change to grab that.

dataset = pd.read_csv('salary.csv')
X = dataset.iloc[:, :-1].values
y = dataset.iloc[:, 1].values

Now X is a matrix of features (our independent variable) and y is a vector of the dependent variable. Perfect!

It's time to split our data into a training set and a test set. Normally, we would do an 80/20 split for our training and testing data. Here, though, we're working with a small dataset of only 30 observations. Maybe this time we'll split up our data so that we have 20 training observations and a test size of 10.

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 1/3, random_state = 0)

You have an X_train, X_test, y_train, and y_test! You're ready to go! (Never forget that there are about a million things to learn about, change, and improve at every step of this process. The power of your model depends on you and everything that you put into it!)

We set a random state of 0 so that we can all get the same result. (There can be random factors in calculations, and I want to make sure we're all on the same page so that nobody gets nervous.)

We'll train our model on the training set and then later predict the results based on our information. Our model will learn the correlations on the training set. Then we will test what it learned by having it predict values with our test set. We can compare our results with the actual results on the test set to see how our model is doing!

A lways split your data into training and testing sets! If you test your results on the same data you used to train it, you'll probably have really great results, but your model isn't good! It just memorized what you wanted it to do, rather than learning anything that it can use with unknown data. That's called overfitting, and it means that you did not build a good model!

Feature scaling

We actually don't need to do any feature scaling here!

Linear regression

Now we can fit the model to our training set!

We'll use Scikit-learn learn for this. First, we'll import the linear model library and the linear regression class. Then we'll create an object of the class - the regressor. We'll use a method (the fit method) to fit the regressor object that we create to the training set. To create the object, we name it, then call it using the parenthesis. We can do all of that in about three lines of code!

Let's import linear regression from Scikit-Learn so that we can go ahead and use it. Between the parenthesis, we'll specify which data we want to use so our model knows exactly what we want to fit. We want to grab both X_train and y_train because we're working with all of our training data.

You can look at the documentation if you want more details!

Now we're ready to create our regressor and fit it to our training data.

from sklearn.linear_model import LinearRegression
regressor = LinearRegression()
regressor.fit(X_train, y_train)

There it is! We're using simple linear regression on our data and we're ready to try out our predictive ability on our test set!

This is machine learning! We created a machine, the regressor, and we had it learn the correlation between years of experience and salary on the training set.

Now it can predict future data based on the information that it has. Our machine is ready to predict a new employee's salary based on the number of years of experience that the employee has!

Let's use our regressor to predict new observations. We want to see how the machine has learned by looking at what it does with new observations.

We'll create a vector of predicted values. This is a vector of predictions of dependent variables that we'll call y_pred. To do this, we can take the regressor we created and trained and use the predict method. We need to specify which predictions to make, so we want to make sure we include the test set. For our input parameter in regressor.predict, we want to specify the matrix of features of new observations, so we'll specify X_test.

y_pred = regressor.predict(X_test)

Seriously. That takes a single line of code!

Now y_test are the real salaries of the 10 observations in the test set and y_pred are the predicted salaries of these 10 employees predicted by our model.

You did it! Linear regression in four lines of code!

Visualization

Let's visualize the results! We need to see what the difference is between our predictions and the actual results.

We can plot the graphs in order to interpret the result. First, we can plot the real observations using plt.scatter to make a scatter plot. (We imported matplotlib.pyplot earlier as plt).

We'll look at the training set first, so we'll plot X_train on the X coordinates and y_train on y coordinates. Then we probably want some color. We'll do our observations in blue, and our regression line (predictions) in red. For the regression line we'll use X_train again for the X coordinates, and then the predictions of the X_train observations.

Let's also fancy it up a little with a title and labels for the x-axis and y-axis.

plt.scatter(X_train, y_train, color = 'blue')
plt.plot(X_train, regressor.predict(X_train), color = 'red')
plt.title('Salary vs Experience (Training set)')
plt.xlabel('Years of Experience')
plt.ylabel('Salary')
plt.show()

Now we can see our blue points, which are our real values and our predicted values along the red line!

Let's do the same for the test set! We'll change the test set title and change our "train" to "test" in the code.

plt.scatter(X_test, y_test, color = 'blue')
plt.plot(X_train, regressor.predict(X_train), color = 'red')
plt.title('Salary vs Experience (Test set)')
plt.xlabel('Years of Experience')
plt.ylabel('Salary')
plt.show()

Make sure you notice that we aren't changing X_train to X_test in the second line. Our regressor is already trained by the training set. When we trained, we obtained one unique model equation. If we replace it, we'll obtain the same line and we'll probably build new points of the same regression line.

This is a pretty good model!

Our model is doing a nice job of predicting these new employee salaries. Some of the actual observations are the same as the predictions, which is great. There isn't a 100% dependency between the y and X variables, so some of the predictions won't be completely accurate.

You did it! You imported libraries, cleaned and preprocessed data, built and trained a simple linear regressor, used it to make predictions, and you even visualized the results!

Congratulations!!!

Want more?

Multiple linear regression is up next!

Keep learning!

Machine learning is built on statistics and you can't begin to understand machine learning without concepts like the simple linear regressor. But that doesn't mean that statistics and machine learning are the same things! A linear regressor is very much a tool of statistics (and data science), in addition to being a part of the basic building blocks of machine learning.

If you're interested in learning more about the differences between statistics and machine learning, I recommend you check out this fantastic piece by Matthew Stewart, PhD Researcher! He does a beautiful job of clarifying these concepts and I highly recommend you take some time to read through his article.

As always, if you're doing anything cool with this information, let people know about it in the responses below or reach out any time on LinkedIn @annebonnerdata!

You might want to check out some of these articles too:

Thanks for reading!

# data[].attributes.attachment_link.title
The complete beginner's guide to machine learning: simple linear regression in four lines of code!

# data[].attributes.attachment_link.description
A clear and comprehensive blueprint for absolutely anyone who wants to build a simple machine learning model Even you can build a machine learning model. Seriously! Good data alone doesn't always tell the whole story. Are you trying to figure out what someone's salary should be based on their years of experience?

# data[].attributes.attachment_link.readable_text
I've been programming in Object Oriented languages for decades. The first OO language I used was C++ and then Smalltalk and finally .NET and Java.

I was gung-ho to leverage the benefits of Inheritance, Encapsulation, and Polymorphism. The Three Pillars of the Paradigm.

I was eager to gain the promise of Reuse and leverage the wisdom gained by those who came before me in this new and exciting landscape.

I couldn't contain my excitement at the thought of mapping my real-world objects into their Classes and expected the whole world to fall neatly into place.
I couldn't have been more wrong.
Inheritance, the First Pillar to Fall

At first glance, Inheritance appears to be the biggest benefit of the Object Oriented Paradigm. All the simplistic examples of shape hierarchies that are paraded out as examples to the newly indoctrinated seem to make logical sense.

And Reuse is the word of the day. No... make that the year and perhaps evermore.

I swallowed this whole and rushed out into the world with my newfound insight.

Banana Monkey Jungle Problem

With religion in my heart and problems to solve, I started building Class Hierarchies and writing code. And all was right with the world.

I'll never forget that day when I was ready to cash in on the promise of Reuse by inheriting from an existing class. This was the moment I had been waiting for.

A new project came along and I thought back to that Class that I was so fond of in my last project.

No problem. Reuse to the rescue. All I gotta do is simply grab that Class from the other project and use it.

Well... actually... not just that Class. We're gonna need the parent Class. But... But that's it.

Ugh... Wait... Looks like we gonna also need the parent's parent too... And then... We're going to need ALL of the parents. Okay... Okay... I handle this. No problem.

And great. Now it won't compile. Why?? Oh, I see... object contains object. So I'm gonna need that too. No problem.

Wait... I don't just need that object. I need the object's parent and its parent's parent and so on and so on with every contained object and ALL the parents of what those contain along with their parent's, parent's, parent's...

Ugh.

There's a great quote by Joe Armstrong, the creator of Erlang:

The problem with object-oriented languages is they've got all this implicit environment that they carry around with them. You wanted a banana but what you got was a gorilla holding the banana and the entire jungle.

Banana Monkey Jungle Solution

I can tame this problem by not creating hierarchies that are too deep. But if Inheritance is the key to Reuse, then any limits I place on that mechanism will surely limit the benefits of Reuse. Right?

Right.

So what's a poor Object Oriented Programmer, who's had a healthy helping of the Kool-aid, to do?

Contain and Delegate. More on this later.

The Diamond Problem

Sooner or later, the following problem will rear its ugly and, depending on the language, unsolvable head.

Most OO languages do not support this, even though this seems to make logical sense. What's so difficult about supporting this in OO languages?

Well, imagine the following pseudocode:

Class PoweredDevice {
}

Class Scanner inherits from PoweredDevice {
 function start() {
 }
}

Class Printer inherits from PoweredDevice {
 function start() {
 }
}

Class Copier inherits from Scanner, Printer {
}

Notice that both the Scanner class and the Printer class implement a function called start.

So which start function does the Copier class inherit? The Scanner one? The Printer one? It can't be both.

The Diamond Solution

The solution is simple. Don't do that.

Yes that's right. Most OO languages don't let you do this.

But, but... what if I have to model this? I want my Reuse!

Then you must Contain and Delegate.

Class PoweredDevice {
}

Class Scanner inherits from PoweredDevice {
 function start() {
 }
}

Class Printer inherits from PoweredDevice {
 function start() {
 }
}

Class Copier {
 Scanner scanner
 Printer printer
 function start() {
 printer.start()
 }
}

Notice here that the Copier class now contains an instance of a Printer and of a Scanner. It delegates the start function to the Printer class's implementation. It could just as easily delegated to the Scanner.

This problem is yet another crack in the Inheritance pillar.

The Fragile Base Class Problem

So I'm making my hierarchies shallow and keeping them from being cyclical. No diamonds for me.

And all was right with the world. That is until...

One day, my code works and the next day it stops working. Here's the kicker. I didn't change my code.

Well, maybe it's a bug... But wait... Something did change ...

But it wasn't in my code. Turns out the change was in the class that I inherited from.

How could a change in the Base class break my code??

This is how...

Imagine the following Base class (It's written in Java, but it should be easy to understand if you don't know Java):

import java.util.ArrayList;

public class Array
{
 private ArrayList<Object> a = new ArrayList<Object>();

 public void add(Object element)
 {
 a.add(element);
 }

 public void addAll(Object elements[])
 {
 for (int i = 0; i < elements.length; ++i)
 a.add(elements[i]); // this line is going to be changed
 }
}

IMPORTANT: Notice the commented line of code. This line is going to be changed later which will break things.

This class has 2 functions on its interface, and . The function will add a single element and will add multiple elements by calling the add function.

And here's the Derived class:

The ArrayCount class is a specialization of the general class. The only behavioral difference is that the ArrayCount keeps a of the number of elements.

Let's look at both of these classes in detail.

And all works fine.

Now for the breaking change. The commented line of code in the Base class is changed to the following:

  public void addAll(Object elements[])
 {
 for (int i = 0; i < elements.length; ++i)
 add(elements[i]); // this line was changed
 }

As far as the owner of the Base class is concerned, it still functions as advertised. And all of the automated tests still pass.

But the owner is oblivious to the Derived class. And the owner of Derived class is in for a rude awakening.

Now ArrayCount addAll() calls its parent's which internally calls the which has been OVERRIDEN by the Derived class.

This causes the to be incremented each time the Derived class's is called and then it's incremented AGAIN by the number of elements that were added in the Derived class's .
IT'S COUNTED TWICE.
If this can happen, and it does, the author of the Derived class must KNOW how the Base class has been implemented. And they must be informed about every change in the Base class since it could break their Derived class in unpredictable ways.

Ugh! This huge crack is forever threatening the stability of precious Inheritance pillar.

The Fragile Base Class Solution

Once again Contain and Delegate to the rescue.

By using Contain and Delegate, we go from White Box programming to Black Box programming. With White Box programming, we have to look at the implementation of the base class.

With Black Box programming, we can be completely ignorant of the implementation since we cannot inject code into the Base class by overriding one of its functions. We only have to concern ourselves with the Interface.

This trend is disturbing...

Inheritance was supposed to be a huge win for Reuse.

Object Oriented languages don't make Contain and Delegate easy to do. They were designed to make Inheritance easy.

If you're like me, you're starting to wonder about this Inheritance thing. But more important, this should shake your confidence in the power of Classification via Hierarchies.

The Hierarchy Problem

Every time I start at a new company, I struggle with the problem when I'm creating a place to put my Company Documents, e.g. the Employee Handbook.

Do I create a folder called Documents and then create a folder called Company in that?

Or do I create a folder called Company and then create a folder called Documents in that?

Both work. But which is right? Which is best?

The idea of Categorical Hierarchies was that there were Base Classes (parents) that were more general and that Derived Classes (children) were more specialized versions of those classes. And even more specialized as we make our way down the inheritance chain. (See the Shape Hierarchy above)

But if a parent and child could arbitrarily switch places, then clearly something is wrong with this model.

The Hierarchy Solution

What's wrong is...

Categorical Hierarchies don't work.

So what are hierarchies good for?

Containment.

If you look at the real world, you'll see Containment (or Exclusive Ownership) Hierarchies everywhere.

What you won't find is Categorical Hierarchies. Let that sink in for a moment. The Object Oriented Paradigm was predicated upon the real world, one filled with Objects. But then it uses a broken model, viz. Categorical Hierarchies, where there is no real-world analogy.

But the real world is filled with Containment Hierarchies. A great example of a Containment Hierarchy is your socks. They are in a sock drawer which is contained in one drawer in your dresser which is contained in your bedroom which is contained in your house, etc.

Directories on your hard drive are another example of a Containment Hierarchy. They contains files.

So how do we categorize then?

Well, if you think of the Company Documents, it pretty much doesn't matter where I put them. I can put them in a folder of Documents or a folder called Stuff.

The way I categorize it is with tags. I tag the file with the following tags:

Document
Company
Handbook

Tags have no order or hierarchy. (This solves the Diamond Problem too.)

Tags are analogous to interfaces since you can have multiple types associated with the document.

But with so many cracks, it looks like the Inheritance pillar has fallen.
Goodbye, Inheritance.
Encapsulation, the Second Pillar to Fall

At first glance, Encapsulation appears to be second biggest benefit of Object Oriented Programming.

Object state variables are protected from outside access, i.e. they're Encapsulated in the Object.

No longer will we have to worry about global variables that are being accessed by who-knows-who.

Encapsulation is a Safe for your variables.

This Encapsulation thing is INCREDIBLE!!

Long live Encapsulation...

That is until...

The Reference Problem

For efficiency sake, Objects are passed to functions NOT by their value but by reference.

What that means is that functions will not pass the Object, but instead pass a reference or pointer to the Object.

If an Object is passed by reference to an Object Constructor, the constructor can put that Object reference in a private variable which is protected by Encapsulation.

But the passed Object is NOT safe!

Why not? Because some other piece of code has a pointer to the Object, viz. the code that called the Constructor. It MUST have a reference to the Object otherwise it couldn't pass it to the Constructor?

The Reference Solution

The Constructor will have to Clone the passed in Object. And not a shallow clone but a deep clone, i.e. every object that is contained in the passed in Object and every object in those objects and so on and so on.

So much for efficiency.

And here's the kicker. Not all objects can be Cloned. Some have Operating System resources associated with them making cloning useless at best or at worst impossible.

And EVERY single mainstream OO language has this problem.
Goodbye, Encapsulation.
Polymorphism, the Third Pillar to Fall

Polymorphism was the redheaded stepchild of the Object Oriented Trinity.

It's sort of the Larry Fine of the group.

Everywhere they went he was there, but he was just a supporting character.

It's not that Polymorphism isn't great, it's just that you don't need an Object Oriented language to get this.

Interfaces will give you this. And without all of the baggage of OO.

And with Interfaces, there isn't a limit to how many different behaviors you can mix in.

So without much ado, we say goodbye to OO Polymorphism and hello to interface-based Polymorphism.

Broken Promises

Well, OO sure promised a lot in the early days. And these promises are still being made to naive programmers sitting in classrooms, reading blogs and taking online courses.

It's taken me years to realize how OO lied to me. I too was wide-eyed and inexperienced and trusting.

And I got burned.
Good-bye, Object Oriented Programming.
So then what?

Hello, Functional Programming. It's been so nice to work with you over the past few years.

Just so you know, I'm NOT taking any of your promises at face value. I'm going to have to see it to believe it.
Once burned, twice shy and all.
You understand.
If you liked this, click the💚 below so other people will see this here on Medium.
If you want to join a community of web developers learning and helping each other to develop web apps using Functional Programming in Elm please check out my Facebook Group, Learn Elm Programming https://www.facebook.com/groups/learnelm/

My Twitter: @cscalfani

# data[].attributes.attachment_link.description
I've been programming in Object Oriented languages for decades. The first OO language I used was C++ and then Smalltalk and finally .NET and Java. I was gung-ho to leverage the benefits of Inheritance , Encapsulation , and Polymorphism . The Three Pillars of the Paradigm.

# data[].attributes.attachment_link.readable_text
Everything You Need To Know In One Article

This article aims to outline all of the key points of Python programming language. My target is to keep the information short, relevant and focus on the most important topics which are absolutely required to be understood.

After reading this blog, you will be able to use any Python library or implement your own Python packages.
You are not expected to have any prior programming knowledge and it will be very quick to grasp all of the required concepts. I will also highlight top discussion questions that people usually query regarding Python programming language.
Lets build the knowledge gradually and by the end of the article, you will have a thorough understanding of Python.
Please let me know whether you want me to post exercises and their solutions to help you practice Python.
This article contains 25 key topics. Let's Start.

1. Introducing Python

What Is Python?

Interpreted high-level object-oriented dynamically-typed scripting language.

Python interpreter reads one line of code at a time, interprets it into low level machine language (byte code) and then executes it.

As a result, run time errors are usually encountered.

Why Python?

Python is the most popular language due to the fact that it's easier to code and understand it.

Python is object oriented programming language and can be used to write functional code too.

It is a suitable language that bridges the gaps between business and developers.

Subsequently, it takes less time to bring a Python program to market compared to other languages such as C#/Java.

Additionally, there are a large number of python machine learning and analytical packages.

A large number of communities and books are available to support Python developers.

Nearly all type of applications, ranging from forecasting analytical to UI, can be implemented in Python.

There is no need to declare variable types. Thus it is quicker to implement a Python application.

Why Not Python?

Python is slower than C++, C#, Java. This is due to the lack of Just In Time optimisers in Python.

Python syntactical white-space constraint makes it slightly difficult to implement for new coders.

Python does not offer advanced statistical features as R does.

Python is not suitable for low-level system and hardware interaction.

How Does Python Work?

This image illustrates how python runs on our machines:

The key here is the Interpreter that is responsible for translating high level Python language to low level machine language.

2. Variables - Object Types And Scope

Variables store information that can be used in your program such as to hold user inputs, local states of your program etc.

Variables are also known as names in Python.

Python supports numbers, strings, sets, lists , tuples and dictionaries. These are the standard data types.
Declare And Assign Value To Variable

Assignment sets a value to a variable:

myFirstVariable = 1
mySecondVariable = 2
myFirstVariable = "Hello You"

Assigning a value is known as binding in Python.
Note how I assigned an integer value of 1 and then a string value of "Hello You" to the myFirstVariable variable. This is possible due to the fact that the data types are dynamically typed in python.
Integers, decimals, floats are supported.

Numeric

Longs are also supported. They have a suffix of L e.g. 9999999999999L

value = 1 #integer
value = 1.2 #float with a floating point

Textual information. Strings are sequence of letters.

String value is enclosed in quotation marks:

Strings

Strings are immutable. Once they are created, they cannot be changed e.g.

name = 'farhad'

When string variables are assigned a new value then internally, Python creates a new object to store the value.

It will throw a Type Error

Variables can have local or global scope.

Variables declared within a for/while loop or within a function, as an example, can only exist within the block.

Once the block exists, the variables also become inaccessible.

Local Scope

def some_funcion():
 TestMode = False

print(TestMode) <- Breaks as the variable doesn't exist outside

Variables that can be accessed from any function have a global scope. They exist in the __main__ frame.

You can declare a global variable outside of functions. It's important to note that to assign a global variable a new value, you will have to use the "global" keyword:

Global Scope
Removing the line "global TestMode" will only set the variable to False within the some_function() function. Note: Although I will write more on the concept of modules later, but if you want to share a global variable across a number of modules then you can create a shared module file e.g. configuration.py and locate your variable there. Finally, import the shared module in your consumer modules.
some_function()
print(TestMode) <--Returns False

If you want to find type of a variable, you can implement:

Finding Variable Type

type('farhad')
--> Returns <type 'str'>

Commas are treated as sequence of variables e.g.

Comma In Integer Variables

9,8,7 is three numeric variables

Allows us to perform computation on variables

3. Operations

Python supports basic *, /, +, -

Python also supports floor division

Numeric Operations

Additionally, python supports exponentiation via ** operator:

1//3  #returns 0.333...
1/3 #returns 0 as both of the operands are integers

Python supports Modulus (remainder) operator too:

2**3 = 2 * 2 * 2 = 8

7%2 = 1
Concat Strings:
String Operations
Repeat String:
'A' + 'B' = 'AB'
Slicing: Reversing: Negative Index:
'A'*3 will repeat A three times:  AAA

If you want to start from the last character then use negative index.

y = 'abc'
print(y[:-1]) # will return c
Finding Index
Also used to remove any new line carriages/spaces.

name = 'farhad'
index = name.find('r')

#returns 2

name = 'farhad'
index = name.find('a', 2) # finds index of second a
For Regex, use:
#returns 4

Casting

Set is an unordered data collection. We can define a set variable as:

Set Operations
Intersect Sets
To get what's common in two sets

a = {1,2,3}
Difference In Sets
To retrieve the difference between two sets:

a = {1,2,3}
b = {3,4,5}
c = a.intersection(b)
Union Of Collections
To get a distinct combined set of two sets

a = {1,2,3}
b = {3,4,5}
c = a.difference(b)

Used to write conditional statements in a single line.
Syntax: [If True] if [Expression] Else [If False]
Ternary Operator

For example:

Received = True if x = 'Yes' else False

4. Comments

Single Line Comments

#this is a single line comment

Multiple Line Comments

One can use:

```this is a multi
line
comment```

5. Expressions

Expressions can perform boolean operations such as:

6. Pickling

Converting an object into a string and dumping the string into a file is known as pickling. The reverse is known as unpickling.

Functions are sequence of statements which you can execute in your code. If you see repetition in your code then create a reusable function and use it in your program.

Functions can also reference other functions.

Functions eliminate repetition in your code. They make it easier to debug and find issues.

Finally, functions enable code to be understandable and easier to manage.

In short, functions allow us to split a large application into smaller chunks.

7. Functions

Define New Function

def my_new_function():
 print('this is my new function')

Calling Function

my_new_function()

Finding Length Of String

Call the len(x) function

len('hello')
prints 5

Arguments

def my_new_function(my_value):
 print('this is my new function with ' + my_value)

We can pass in optional arguments by providing a default value to an argument:

def my_new_function(my_value='hello'):
 print(my_value)

#Calling
my_new_function() => prints hello
my_new_function('test') => prints test

If your function can take in any number of arguments then add a * in front of the parameter name:

def myfunc(*arguments):
 return a

myfunc(a)
myfunc(a,b)
myfunc(a,b,c)

It allows you to pass varying number of keyword arguments to a function.

You can also pass in dictionary values as keyword arguments.

Functions can return values such as:

Return

If a function is required to return multiple values then it's suitable to return a tuple (comma separated values). I will explain tuples later on:

def my_function(input):
 return input + 2

resultA, resultB = get_result()
Lambda
get_result() can return ('a', 1) which is a tuple

my_lambda = lambda x,y,z : x - 100 + y - z

my_lambda(100, 100, 100) # returns 0
variable = lambda arguments: expression Lambda functions can be passed as arguments to other functions. dir() and help()

dir() -displays defined symbols

help() - displays documentation

Syntax:

8. Modules

This environment variable indicates where the Python interpreter needs to navigate to locate the modules. PYTHONHOME is an alternative module search path.

What is a module?

If you have a file: MyFirstPythonFile.py and it contains multiple functions, variables and objects then you can import the functionality into another class by simply doing:

PYTHONPATH

Internally, python runtime will compile the module's file to bytes and then it will run the module code.

If you want to import everything in a module, you can do:

How To Import Modules?

If your module contains a function or object named my_object then you will have to do:

import MyFirstPythonFile
Note: If you do not want the interpreter to execute the module when it is loaded then you can check whether the __name__ == '__main__' 2. From
If you only want to access an object or parts of a module from a module then you can implement:

import my_module

This will enable you to access your object without referencing your module:

print(my_module.my_object)

We can also do from * to import all objects

from my_module import my_object
Note: Modules are only imported on the first import. If you want to use C module then you can use PyImport_ImportModule Use import over from if we want to use the same name defined in two different modules.
print(my_object)

from my_module import *

In the example above, packageroot is the root folder. packagefolder is the subfolder under packageroot. my_module is a module python file in the packagefolder folder.

Additionally, the name of the folder can serve as the namespace e.g.

9. Packages
Note: Ensure each directory within your package import contains a file __init__.py. Feel free to leave the files blank. As __init__.py files are imported before the modules are imported, you can add custom logic such as start service status checks or to open database connections etc.
from packageroot.packagefolder.mod import my_object

PIP is a Python package manager.

Use PIP to download packages:

from data_service.database_data_service.microsoft_sql.mod

PIP
Note how Checking Types colons and indentations You can also add conditional logic in the else part. This is known as nested condition. are used to express the conditional logic.
pip install package_name

10. Conditions

Provide a condition and run the loop until the condition is met:

if not isinstance(input, int):
 print 'Expected int'
 return None

#let's write conditions within else
else:
 if a = 2:
 print 'within if of else'
 else:
 print 'within else of else'

Loop for a number of times

11. Loops

Loop over items or characters of a string

While
One-Liner For
while (input < 0):
 do_something(input)
 input = input-1

For
Yielding

Let's assume your list contains a trillion records and you are required to count the number of even numbers from the list. It will not be optimum to load the entire list in the memory. You can instead yield each item from the list.

xrange instead of range can be used to iterate over the items.

for  i in range(0,10)

Let's do a simple exercise to find if a character is in two words

for letter in 'hello'
 print letter

Syntax:

[Variable] AggregateFunction([Value] for [item] in [collection])

If you want to end the loop

Combine For with If

Let's write Fibonacci for loop:

name = 'onename'
anothername = 'onenameonename'

A function calling itself is known as recursion.
Let's implement a factorial recursive function:
for character in name:
 if character in anothername
 print character

0! = 1 #Factorial of 0 is 1

n! = n(n-1)! #Factorial of n is n * factorial of n-1

Steps:

Create a function called factorial with input n

If n = 0 return 1 else do n x factorial of n-1

Break
Another Example: Let's write Fibonacci recursive function: Rules:

First two digits are 0 and 1

Rest add last two digits

for i in range(0,10):
 if (i==5):
 break
Steps:
12. Recursion

Rules:
It is important to have an exit check otherwise the function will end up in infinite loop.
def factorial(n):
 if n==0:
 return 1
 else:
 return n * factorial(n-1)

Python code is loaded into frames which are located into a Stack.

Functions are loaded in a frame along with the parameters and variables.

Subsequently, frames are loaded into a stack in the right order of execution.

Stack outlines the execution of functions. Variables that are declared outside of functions are stored in __main___

Stacks executes the last frame first.

0, 1, 1, 2, 3, 5, 8...

def fibonacci(n):
 if (n<=1):
 return n
 else:
 return fibonacci(n-1)+fibonacci(n-2)

print(fibonacci(6))

13. Frames And Call Stack

To add/update/delete an item, use index:

You can use traceback to find the list of functions if an error is encountered.

Addition, repetition and slices can be applied to lists (just like strings).

List also supports sorting:

14. Collections

Lists

Tuples are like lists in the sense that they can store a sequence of objects. The objects, again, can be of any type.

Tuples are faster than lists.

These collections are indexed by integers.

Tuples are immutable (non-update-able)

Note: If a tuple contains a list of items then we can modify the list. Also if you assign a value to an object and you store the object in a list and then change the object then the object within the list will get updated.
my_list = ['A', 'B']

Dictionary is one of the most important data structure in programming world. It stores key/value pair objects.

It has many benefits e.g. optimised data retrieval functionality.

my_list.append('C') #adds at the end
my_list[1] = 'D' #update
my_list.pop(1) # removes
or
del my_list[1:2] # removes
my_list.extend(another_list) # adds second list at end

You can also create a dictionary as:

my_list.sort()

Tuples

Values of a dictionary can be of any type including strings, numerical, boolean, lists or even dictionaries.

Dictionaries

my_dictionary = dict()
my_dictionary['my_key'] = 1
my_dictionary['another_key'] = 2
Note: If you want to perform vectorised/matrix operations on a list then use NumPy Python package
my_dictionary = {'my_key':1, 'another_key':2}

for key in dictionary:
 print key, dictionary[key]

Allows compilation of new extensions without any error

dictionary.items() # returns items

Once the extensions are compiled, they can be linked.

#checking if a key exists in a dictionary
if ('some key' in dictionary):
 #do something

15. Compilation And Linking

Compilation:

Filter out values based on a condition

Linking:

Applies a computation on each value of a collection. It maps one value to another value e.g. Convert Text To Integers

16. Iterators

Reduces a collection of values to one single value (or a smaller collection) e.g. sum of a collection. It can be iterative in nature.

Iterators

Filter

Map

Reduce

An instance of a class is known as an object. The objects are mutable and their properties can be updated once the objects are created.
Note: If a tuple (immutable collection) contains a list (mutable collection) of items then we can modify the list. Also if you assign a value to an object and you store the object in a list and then change the object then the object within the list will get updated.
Zip

__init__ function is present in all classes. It is executed when we are required to instantiate an object of a class. __init__ function can take any properties which we want to set:

17. Object Oriented Design - Classes

class MyClass:
 def MyClassFunction(self): #self = reference to the object
 return 5

#Create new instance of MyClass and then call the function
m = MyClass()
returned_value = m.MyClassFunction()
Note: self parameter will contain the reference of the object, also referred to as "this" in other programming languages such as C#
__init__

Returns stringified version of an object when we call "print":

class MyClass:
 def __init__(self, first_property):
 self.first_property = first_property

Therefore, __str__ is executed when we execute print.

   def MyClassFunction(self):
 return self.first_property

r will be 123

Methods of an object can be overloaded by providing more arguments as an instance.

We can also overload an operator such as + by implementing our own implementation for __add__

__str__

m = MyClass(123)
print m #Calls __str__

This will result in shallow copy as the reference pointers of the properties will be copied.

Therefore, if one of the properties of the object is an object reference then it will simply point to the same reference address as the original object.

As a result, updating the property in the source object will result in updating the property in the target object.

Therefore shallow copy copies reference pointers.

Fortunately, we can utilise deep-copy:

__cmp__

If MyClass contains a property that references MyOtherClass object then the contents of the property will be copied in the newly created object via deepcopy.

Therefore, deep copy makes a new reference of the object.

Overloading

Shallow Vs Deep Copy Of Objects

import copy
m = MyClass(123)
mm = copy.copy(m)

Consequently, both subclasses will contain the function my_function().

Inheritance can encourage code re-usability and maintenance.

Note: Python supports multiple inheritance unlike C#
Use Python's abc module to ensure all subclasses contain the required features of the Abstract Base Class.
Multi-Inheritance:
import copy
m = MyClass(123)
mm = copy.deepcopy(m)

If you want to call parent class function then you can do:

18. Object Oriented Design - Inheritance

class ParentClass:
 def my_function(self):
 print 'I am here'

All of the objects in Python are stored in a heap space. This space is accessible to the Python interpreter.

Python has an in-built garbage collection mechanism.

It means Python can allocate and de-allocate the memory for your program automatically such as in C++ or C#.

It's responsibility is to clear the space in memory of those objects that are not referenced/used in the program.

class SubClass1(ParentClass):
class SubClass2(ParentClass):

1. Reference counting: Count the number of items an object is referenced, deallocate an object if its count is 0.

2. The second mechanism takes care of circular references, also known as cyclic references, by only de-allocating the objects where the allocation - deallocation number is greater than threshold.

New objects are created in Generation 0 in python. They can be inspected by:

Manual garbage collection can be performed on timely or event based mechanism.

class A(B,C):  #A implments B and C

super(A, self).function_name()

Use the raw_input() function

19. Garbage Collection - Memory Management

As multiple objects can share memory references, python employs two mechanisms:

Use a with/as statement to open and read a file. It is equivalent to using statement in C#.

with statement can take care of closing of connections and other cleanup activities.

Open files
import gc
collected_objects = gc.collect()

20. I/O
Note: readline() can also be executed to read a line of a file. To open two files
From Keyboard
Writing files
user_says = raw_input()
print(user_says)
Note: Use os and shutil modules for files. Note: rw - read-write mode and a - append mode.
Files
Open a connection
with open(file path, 'r') as my_file:
 for line in my_file
Execute a SQL statement
#File is closed due to with/as

with open(file path) as my_file, open(another path) as second_file:
 for (line number, (line1, line2)) in enumerate(zip(my_file, second_file):
To query a rest service
with open(file path, 'w') as my_file:
 my_file.write('test')
To Serialise and Deserialise JSON
SQL

import MySQLdb
database = MySQLdb.connect("host"="server", "database-user"="my username", "password"="my password", "database-name"="my database")
cursor = database.cursor()

cursor.fetch("Select * From MyTable")
database.close()

Web Services

import requests
url = 'http://myblog.com'
response = requests.get(url).text

If you want to raise exceptions then use the raise keyword:

Deserialise:

To catch exceptions, you can do:

If you want to catch specific exceptions then you can do:

finally part of the code is triggered regardless, you can use finally to close the connections to database/files etc.

If you want to use try/catch/finally then you can do:

Serialise:

We can also assign exception to a variable by doing:

If you want to define user-defined constraints then use assert:
Note: Python supports inheritance in exceptions
import json
a = "1"
json_a = json.dumps(a)

21. Error Handling

Raise Exceptions

GIL is Global Interpreter Lock.

It ensures that the threads can execute at any one time and allows CPU cycles to pick the required thread to be executed.

GIL is passed onto the threads that are currently being executed.

Python supports multi-threading.

Note: GIL adds overheads to the execution. Therefore, be sure that you want to run multiple threads.
Catching Exceptions

Decorators can add functionality to code. They are essentially functions that call other objects/functions. They are callable functions - therefore they return the object to be called later when the decorated function is invoked.

Think of decorates that enable aspect-oriented programming

We can wrap a class/function and then a specific code will be executed any time the function is called.

We can implement generic logic to log, check for security checks etc and then attribute the method with the @property tag.

try:
 do_something()
except TypeError:
 print('exception')
finally:
 close_connections()

There are a number of unit testing and mocking libraries available in Python.

An example is to use unittest:

Try/Except/Else

try:
 do something
except IfTryRaisedException1:
 do something else
except (IfTryRaisedException2, IfTryRaisedException3)
 if exception 2 or 3 is raised then do something
else:
 no exceptions were raised

assert <bool>, 'error to throw'

You can create your own exception class by:

class MyException(Exception): pass

22. Multi-Threading And GIL

Simple to code and learn

Object oriented programming language

Great Analytics and ML packages

Faster to develop and bring my solution to market

Offers in-built memory management facilities

Huge community support and applications available

No need to compile as it's an interpreted language

Dynamically typed - no need to declare variables

23. Decorators

24. Unit Testing In Python

Spyder, PyCharm. Additionally various notebooks are used e.g. Jupyter

1.Assume your function simply decrements an input by 1

There are a large number of must-use packages:

def my_function(input):
 return input - 1

2. You can unit test it by:

For Unix: Make script file mode Executable and first line must be:

We can also use doctest to test code written in docstrings.

25. Top Python Discussion Questions

Why Should I Use Python?

How To Make Python Run Fast?

A large number of rich statistical libraries have been written in R

One can execute R code within Python by using Rpy2 python package or by using beaker notebook or IR kernel within Juputer.

Which IDEs Do People Use?

We can use PyChecker and PyLink to catch errors before running the code.

What Are The Top Python Frameworks And Packages?

PyUnit (unit testing), PyDoc (documentation), SciPy (algebera and numerical), Pandas (data management), Sci-Kit learn (ML and data science), Tensorflow (AI), Numpy (array and numerical), BeautifulSoap (web pages scrapping), Flask (microframework), Pyramid (enterprise applications), Django (UI MVVM), urllib (web pages scraping), Tkinter (GUI), mock (mocking library), PyChecker(bug detector), Pylint (module code analysis)
Rest, just practice as much as possible and you can implement your own library in Python because this article contains all the knowledge you need. If you want me to post Python exercises and solutions then please let me know.
How To Host Python Packages?

#(#!/my account/local/bin/python)

2. You can use command line tool and execute it

3. Use PyPRI or PyPI server

Can Python and R be combined?

Is there a way to catch errors before running Python?

Summary

This article outlined the most important 25 concepts of Python in a short, relevant and focused manner. I genuinely hope it has helped someone get better understanding of Python.

I believe I have concentrated on the must know topics which are absolutely required to be understood.This knowledge is sufficient to write your own python packages in the future or using existing Python packages.

Hope it helps.

# data[].attributes.attachment_link.title
Everything About Python - Beginner To Advance

# data[].attributes.attachment_link.description
This article aims to outline all of the key points of Python programming language. My target is to keep the information short, relevant and focus on the most important topics which are absolutely required to be understood. After reading this blog, you will be able to use any Python library or implement your own Python packages.

# data[].attributes.headline
Automatiza decisiones con Inteligencia Artificial en Excel (instructivo para novatos. Parte 1) 

# data[].attributes.subject
Automatiza decisiones con Inteligencia Artificial en Excel (instructivo para novatos. Parte 1) 

# data[].attributes.attachment_link.readable_text
Seguramente te has creído que la inteligencia artificial ( AI en inglés) es cosa de supergenios, pero no es así. Si eres emprendedor seguro tendrás el inconveniente administrativo de tomar decisiones, y si eres inversor tendrás el inconveniente de que dejas tu dinero en manos de alguien que toma decisiones con tu dinero que puede tomar malas decisiones con o sin intención. ¿Es posible automatizar altos ejecutivos? Sí.

Te aseguro que con este post podrás hacer inteligencias artificiales sencillas sin ser un experto y lo podrás hacer en Excel.

A como te metas en ello hay un modelo de negocios esperando. El de automatizar altos ejecutivos. Ya sé que dicen que es imposible, pero recuerda que hace 30 años era imposible automatizar a una secretaria. A como leas este artículo te convencerás de que podría haber un potencial de negocios en reemplazar altos ejecutivos con ordenadores.

En este post mostraré primero como construir neuronas de manera sencilla usando Excel, en segundo lugar usaré ejemplos simples y en tercer lugar mostraré la arquitectura general, el esquema conceptual de cómo automatizar la labor de los altos ejecutivos de la empresa. Eso se llama igualdad de automatización, pues si vas a automatizar al trabajador de fábrica, también tendrías que poder automatizar al ejecutivo para eventualmente no necesitarlo. Siendo emprendedor te ahorras contratar un consultor, un ayudante conocedor o servicos de asistencia técnica, y si eres inversor te ahorras tener que contratar un alto ejecutivo.

Conceptos básicos de inteligencia artificial
La red neuronal (neuron network)
Una neurona es una célula que recibe información de otras neuronas, hace un procesamiento simple, y luego envía un resultado a otras neuronas. Podríamos decir que una neurona es una función, y casualmente lo que tu escribes en las celdas de Excel suele ser una función. Y así entre un dato de entrada (input data) y la información de salida o resultados (output data), hay funciones donde las salidas de una función son la entrada de otra función. Como puedes ver las neuronas de agrupan en capas.
Machine learning
Una cosa es el funcionamiento de las neuronas, su capacidad de pensar y entregar resultados, y otra cosa es su capacidad de aprender, que modifica su funcionamiento. El concepto de aprender ( machine learning) tiene que ver con ajustar coeficientes de esas funciones.

En este post no entraré en el detalle de la complicación del machine learning, porque este es un campo en empresas y academia están aún aprendiendo, investigando, perfeccionando. En forma muy simplificada podríamos pensar en machine learning como un proceso donde la máquina estará haciéndote la mínima cantidad de preguntas para aprender y ajustar coeficientes para dar los mejores resultados posibles. En lugar de entrar en arquitecturas complicadas, y ajustes de coeficientes a lo largo de toda una red, lo que haré será ofrecer maneras simples (para fines didácticos solamente) en las que tu puedes crear neuronas individuales expertas. Claro, no es lo mismo una neurona experta que una red de neuronas expertas, así como no es lo mismo un músico experto que una orquesta de músicos expertos que no coordinan la sinfonia entre ellos en el concierto. Pero para los casos que veremos esas maneras simples serán trucos suficientes para que tu construyas tus neuronas sin tener que pasar por un machine learning que te quiebre la cabeza.
La estructura de la neurona
Una neurona es una función con una estructura muy particular.

Los datos de entrada son los valores x. Tenemos x1, x2, x3, etc....

Para cada x hay un coeficiente w. Tenemos coeficientes w1, w2, w3, etc... y hay un valor b que se conoce como "error". Cada coeficiente w se entiende como "peso" (¿recuerdas cuando te han hablado de un peso ponderado?)

Tomas cada x y lo multiplicas por el w y lo sumas con el resto de multiplicaciones x por w y finalmente b.

Si recuerdas en secundaria la funcion lineal f(x) = m * x + b, en una neurona m * x + b se entendería esa operación como x1 * w1 + b.

Al resultado de la suma de las x por w más b se le aplica una función y finalmente se obtiene el valor y.

Tu neurona completa se mira así.

Implementando funciones en Excel
y = función (x1 * w1 + x2 * w2 + x3 * w3 +....+ b)

Acabas de abrir Excel y oprimes Alt-F11 para abrir el editor (conocido como IDE).

Haces click derecho en la zona en blanco indicada, seleccionas Insert, luego seleccionas Module.

Luego ingresas el siguiente texto.

Cierras la ventana del IDE y ya estamos listos para usar las 3 funciones.en celdas de Excel.

Puedes probar la función Sigmoid en Excel para ver si funciona.

Recuerda salvar la hoja de Excel con extensión xlsm para que conserve el código de las funciones que insertaste.

Cabe notar que y = función(x1*w1+x2*w2+x3*w3+....+b) en este caso es y = sigmoid(A1) donde la celda A1 es el x1 y w1 tiene valor 1 y b es igual a 0. Si graficamos la función sigmoide se mira lo siguiente:

Tiene una forma de S.

Sus valores se encuentran entre 0 y 1

De un lado es casi 0 y del otro es casi 1.

La transición del 0 al 1 es suave.

Si lograras cerrar el intervalo de transición, y aproximar los valores a 0 y a 1 podrías tener una función prácticamente binaria. Muy útil para convertir algo en valores de falso y verdadero.

Miramos que el 5 y -5 prácticamente es casi cero y en 10 está demasiado cerca de cero.

Las otras funciones se pueden graficar también. La normalizadora permite convertir valores entre 0 y 1 y pasarlos de -1 a 1. La rectificadora tiene valores cero para números negativos y el valor se preserva para números positivos.

Como puedes ver, las conductas de las funciones que intervienen en las AI suelen tener conductas no lineales. Esto hace que el resultado de usarlas sea muy poco intuitivo, pero muy inteligente.

Caso 1: Cálculo del agotamiento de un trabajador

Nos han dado la siguiente caricatura y nos piden que calculemos cuanta energía le queda al trabajador luego de cierta cantidad de horas laborales. Sólo vamos a usar una sola neurona.

Observamos que a partir de la hora 7 la energía empieza a descender y para la 13ava hora ya está agotado.

De todas las funciones que vinos la sigmoide parece tener una transición suave, pero tenemos que hacer una conversión. La función sigmoide pide un valor tque es 1 (100%) en 5 y es casi cero en -5. Entonces tenemos que buscar una manera de convertir horas en números de la curva sigmoide.

Hora 7 equivale a t = 5

Hora 13 equivale a t = -5

Vas a abrir el IDE con Alt F11 y vas a agregar una nueva función, al final de lo que ya habías escrito.

Esa función hace la conversión de horas a valor t.

Como ves en la función tvalue los valores 7 y 5 indican conversión de 7 a 5, y los valores 13 y -5 indican conversión de 13 a -5, y la x que ingresa a la neurona es el valor de la celda A2.

Al aplicar la función sigmoid estamos obteniendo la salida de la neurona.

En este caso la función tvalue nos pide 4 valores para aprender, y un valor x para calcular.

Las equivalencias necesitaban de un trabajo de ajuste de coeficientes que tvalue ya ha hecho. Cabe notar que en este caso los datos 7,5,13,-15 son los datos reales que tenemos. 7 y 13 son datos tomados en la realidad y 5 y -5 son resultados sirven para ajustar coeficientes por medio de tvalue.

En este caso la función tvalue hace el trabajo de encontrar una equivalencia lineal.

Desde un punto de vista hago trampa al usar tvalue como algoritmo de ajuste de coeficientes (machine learning), porque no hay un aprendizaje usando toneladas de datos y usando los algoritmos típicos de machine learning.

Desde otra perspectiva es lo correcto, porque se trata de hacer que la función aprenda con una cantidad mínima de preguntas, eficiente, que es lo que los académicos buscan al final.

Esto nos indica que para la hora 8 el trabajador tiene el 96% de la energía para trabajar. Tiene cerca de un 4% de agotamiento. Su trabajo (output) será menor.

Caso 2: Calculando la mejor jornada laboral para trabajadores

Sabemos que aun trabajador le afecta el cansancio, y también sabemos que la calidad del trabajo disminuye con la velocidad. Muchos jefes y gerentes piden cantidad y calidad.

La ideología dice que más horas al día significa más producción y calidad.

La ideología dice que más horas al día significa más producción y calidad.

Entonces la ideología empuja al gerente y jefe a convertirse en capataz de esclavos.

Bajo la ideología, tener a un trabajador desocupado es dinero perdido.

Hay un coste por hora y de allí sólo hay que exigir más producción y más calidad y todo resuelto. El trabajador eficiente termina y se queda sin hacer nada y se le insta a trabajar, a que no sea vago. Esta ideología sólo maximiza la utilización del trabajador. Pero ¿de verdad aumenta la calidad (quality) y cantidad (output) si tratas a los trabajadores com máquinas? Como en este ejemplo somos unosinhumanos, vamos a tratar a los trabajadores como máquinas, y nos vamos a imaginar que son un motor de submarino nuclear. Los sumbarinos tienen "full speed" (a toda máquina) al 50% de la potencia total del motor. A partir de allí empieza "flank speed" (velocidad de flanco) que va de 50% a 100% y si usas esa potencia debes encender los enfriadores de alta velocidad. Esto quiere decir que no existe eso de que el motor de un 110% como a veces le piden a los trabajadores.

¿Has notado que a los trabajadores humanos les exigen incremento en su desempeño de 20% años a año? A los ordenadores no se les hace eso. Si quieres hardware y software más rápido, debes pagar por ello.

El ordenador se enferma de virus y corren a curarle. Al humano le despiden.

Las empresas contratan soporte 24/7 para las máquinas, pero no para los trabajadores.

Ahora que ya sabemos cómo se porta un motor de submarino digamos que al 50% de potencia del reactor el humano aún da su mejor calidad. A partir de 50% comienza a decaer su calidad. Tenemos la siguiente equivalencia de función sigmoide:

50% de potencia del reactor (a toda máquina) equivale a 100% de calidad.

100% de potencia del reactor (velocidad de flanco) tenemos 0% de calidad porque para el humano no hay enfriadores de alta velocidad y se funde el motor.

Luego de hacer lo mismo que hicimos antes, calculamos tablas para Output vs horas trabajadas (lo que calculamos en el caso 1) y calidad vs potencia de reactor usando sigmoides.

Luego multiplicamos los valores para ver cómo se comporta el ser humano. Esta es la tercera neurona que no tiene funciones extrañas asociadas. Es una fumción muy entendible.

Multiplicador de productividad = Output x Quality

Cabe notar que el resultado es sólo un multiplicador sin unidades que indica capacidad máxima del ser humano. Veamos como se comporta el multiplicador.

Esta tabla nos dice que no parece bueno forzar al ser humano a muchas horas y a velocidad de flanco. Pero esto es sólo capacidad humana. ¿Qué pasa con la producción? Esta es nuestra cuarta neurona, también muy entendible.

Producción (unidades) = Multiplicador de productividad x Capacidad nominal de producción por hora x Cantidad de horas

Supongamos que en 8 horas sacan 100 unidades. Cabe aclarar que este gráfico muestra lo que ocurre cuando la producción es intensiva en horas máquina, y lo que afecta la calidad viene en la inspección.

¿Qué pasaría si es intensiva en mano de obra? Habría que multiplicar por la potencia del reactor.

Producción (unidades) = Multiplicador de productividad x Capacidad nominal de producción por hora x Cantidad de horas x speed

El desempeño de la producción se muestra así. Ojo que hablamos de unidades que cumplen las especificaciones de calidad, que producir mucho se puede, pero va a terminar en desecho o reproceso.

Parece que el óptimo está en 70% de potencia de reactor, y 8 horas. En Japón hay empresas que practican la ideología del karoshi, que literalmente significa "muerte por exceso de trabajo" donde no hay límite de horas.

18 October 2016. Authorities rule Joey Tocnang's death was directly related to the long hours of overtime - up to 122.5 hours a month - he was forced to perform - Death from overwork: Japan's 'karoshi' culture blamed for young man's heart failure

En su mundo de fantasía, los ejecutivos del karoshi imaginan que esta gráfica se comporta de esta manera.

Esto nos dice que con una simple AI de 4 neuronas en Excel mejoramos la capacidad de decisión de un alto ejecutivo japonés. ¿Comprendes ahora el poder de la inteligencia artificial? resulta incluso alarmante que la AI entregue un resultado más humanitario que la decisón de altos ejecutivos. ¿Puede la máquina ser más humana que los humanos? En este caso lo es.

¿Que necesitas aumentar la producción? Entonces contrata más gente. ¿Qué eso significa entrenar gente? Pues sí. La AI ya nos ha dicho donde es que tenemos la máxima productividad con calidad. Si tratas de bailar con la realidad obtienes desempeño inferior y maltratas a la gente.

¿Te imaginas crear una AI y cobrar suscripción por los servicios de la AI a sólo una fracción del precio del salario de un alto ejecutivo? En la próxima entrega hablaré de la estructura dentro de la cual debes montar tus redes neuronales para automatizar a un alto ejecutivo.

¿Qué es lo diferente de la AI respecto del software común? Lo distinto es que antes el software era una receta de cocina que entregaba platos, y la AI es un cerebro que aprende y que entrega resultados inteligentes que resultan de operaciones con funciones no lineales.

Lo que hace superior a las AI es que los altos ejecutivos se "enfocan" mientras que la AI lo ve todo. Los ejecutivos siguen ideologías muy definidas:

La realidad: Una empresa es un grupo de personas que hace cosas.

Ideología financiera: Una empresa es dinero que hace dinero.

Ideología automatizadora: Una empresa es un ordenador que hace dinero.

Ideología política: Un partido es un grupo de ideologías de corto plazo que hace cosas.

El problema de la ideología es que pierde de vista todo lo que se salga del alcance de su ideología. Algo tan simple como la jornada laboral, que cualquiera entiende, no está bajo la cobertura de los ejecutivos. Es que la ideología es simple: "Si A entonces B" una sola neurona lineal sin sigmoides, ni rectificadores, ni nada. Les pagan muchísimo dinero. ¿Cuánto dinero se podría ahorra una empresa si automatiza a los ejecutivos? Esta es una lista de los salarios de ejecutivos en India. Ni siquiera por estar en país barato terminan de ser baratos. ¿Cuánto dinero hay en un modelo de negocios que preste servicios de altos ejecutivos AI?

Infosys ya alega haber destruido 11000 empleos ( Infosys releases 11,000 employees due to automation: Key takeaways from board meet). Se podría agregar algunos más entre altos ejecutivos de empresas de países baratos o caros. ¿Hay potencial de negocios en automatizar a los altos ejecutivos y cobrar por ello?
Finalmente he de agradecer a J. P. Luna por facilitar este material para su publicación.
Si eres emprendedor y automatizas, siempre te quedará el lado de inversor, de lograr sueños que los ejecutivos de negocios establecidos no tienen.

# data[].attributes.attachment_link.title
Automatiza decisiones con Inteligencia Artificial en Excel (instructivo para novatos. Parte 1)

# data[].attributes.attachment_link.description
Seguramente te has creído que la inteligencia artificial es cosa de supergenios, pero no es así. Si eres emprendedor seguro tendrás el inconveniente administrativo de tomar decisiones, y si eres inversor tendrás el inconveniente de que dejas tu dinero en manos de alguien que toma decisiones con tu dinero que puede tomar malas decisiones con o sin intención.

# data[].attributes.attachment_link.readable_text
En el artículo PyTorch, ¿Qué es y cómo se instala? dimos a conocer el paquete PyTorch, un paquete de computación matemática diseñado para trabajar con tensores, está centrado para su utilización en el campo del machine learning, más en concreto en el desarrollo de redes neuronales. En este artículo no vamos a hablar sobre que son las redes neuronales, pero es necesario conocer lo básico de ellas para seguir este tutorial y poder programar así nuestra primera red neuronal con pytorch.

En resumen, en este pequeño tutorial aprenderemos a usar PyTorch para crear, entrenar y predecir con nuestra primera red neuronal.

Primeros pasos

Lo primero que debemos hacer es importar torch y algunos módulos de esta librería:

torch.nn: La librería de redes neuronales que utilizaremos para crear nuestro modelo.

torch.autograd: En concreto el módulo Variable de esta librería que se encarga de manejar las operaciones de los tensores.

torchvision.datasets: El módulo que ayudará a cargar el conjunto de datos que vamos a utilizar y explicaremos más adelante.

torchvision.transforms: Este módulo contiene una serie de funciones que nos ayudarán modificando el dataset.

torch.optim: De aquí usaremos el optimizador para entrenar la red neuronal y modificar sus pesos.

Para ello ejecutamos el siguiente código:

import torch import torch.nn as nn import torch.nn.functional as F from torch.autograd import Variable import torchvision.datasets as dset import torchvision.transforms as transforms import torch.optim as optim

Dataset a utilizar

Una vez importados los módulos, debemos cargar el conjunto de datos que vamos a utilizar, en este tutorial vamos a utilizar el conjunto conocido como MNIST. Se trata de un conjunto de imágenes de dígitos escritos a mano, contiene un total de 60000 imágenes para entrenamiento y 10000 para test. Todos los dígitos están normalizados en tamaño y centrados en la imagen de tamaño 28x28 en escala de grises. El objetivo de esta base de datos es clasificar cada imagen diciendo a que número entre el 0 y el 9 pertenece.

Ejemplo de los dígitos de MNIST ( fuente)

Para cargar de forma rápida el dataset vamos a utilizar el modulo datasets de torchvision, además debemos definir que transformaciones vamos a aplicarle a todas las muestras. En nuestro caso únicamente vamos a pasar el objeto obtenido a Tensor con lo que PyTorch podrá realizar cálculos, torchvision.transforms contiene una gran cantidad de transformaciones para aplicar a los datasets. Es recomendable normalizar nuestro dataset, este dataset viene por defecto normalizado valores entre 0 y 1. También vamos a fijar la semilla del generador aleatorio para poder replicar los experimentos.

torch.manual_seed(123) #fijamos la semilla trans = transforms.Compose([transforms.ToTensor()]) #Transformador para el dataset

Después definimos donde vamos a descargar los datos, en nuestro caso en la carpeta data dentro de la que estamos utilizando. La herramienta datasets de torchvision es muy fácil y cómoda para usar, simplemente debemos elegir el dataset entre los que tiene disponibles, debemos indicar la ruta a la carpeta con el parámetro root, si queremos el dataset de entrenamiento (en caso contrario obtendremos el de test), las transformaciones que se le van a aplicar que hemos definido anteriormente y si queremos descargar el dataset en el caso de que no lo tengamos almacenado ya. Para ello utilizaremos el siguiente código:

root="./data" train_set = dset.MNIST(root=root, train=True, transform=trans, download=True) test_set = dset.MNIST(root=root, train=False, transform=trans)

Lo siguiente que debemos hacer es definir el tamaño de batch, que es simplemente el tamaño de conjunto de datos que vamos a pasar a la red a la vez, es decir, en vez de clasificar y actualizar los pesos muestra por muestra vamos a hacerlo cada conjunto de x muestras, esto ayuda a paralelizar la red pero debemos tener cuidado porque tamaños muy grandes de batch pueden requerir grandes cantidades de memoria RAM, ya sea en la GPU o en la CPU. Después definimos un Dataloader para el dataset que hemos descargado. Un Dataloader no es más que un generador que nos proporcionará las muestras en grupos de un tamaño dado, además contiene ciertas funciones de gran ayuda como la de reordenar aleatoriamente el dataset en cada iteración ( shuffle). El código para generar los dataloader de entrenamiento y test es el siguiente:

batch_size = 128 train_loader = torch.utils.data.DataLoader( dataset=train_set, batch_size=batch_size, shuffle=True) test_loader = torch.utils.data.DataLoader( dataset=test_set, batch_size=batch_size, shuffle=False)

Podemos saber la cantidad de conjuntos (batches) de tamaño 128 tenemos simplemente utilizando el método len de Python. En el caso de que el dataset no sea un múltiplo del tamaño de batch, el último batch será de un tamaño reducido.

print ('Trainning batch number: {}'.format(len(train_loader))) print ('Testing batch number: {}'.format(len(test_loader)))

Creación del modelo y entrenamiento

Una vez tenemos el dataset debemos decidir que topología de red neuronal vamos a utilizar, en nuestro caso vamos a optar por una muy simple con una sola capa oculta que contiene 256 neuronas. Para ello vamos a definir una clase que desciende de nn.Module. En el __init__ definiremos las capas que va a tener la red y la función de pérdida que vamos a utilizar. La primera capa será la capa oculta y recibirá el tensor de entrada con los datos que tienen un tamaño de 28x28, por tanto la capa tendrá un tamaño de entrada de 28*28 y un tamaño de salida de 256. La segunda capa será la capa de salida que recibirá los datos de la capa oculta con un tamaño de 256 y su salida será igual al número de clases, en nuestro caso 10 clases. La función de pérdida que hemos escogido para nuestra red es la entropía cruzada.

Después debemos definir la función forward, es decir, las operaciones que se van a realizar desde la entrada de la red hasta su salida. Aquí vamos a definir las funciones que se aplicarán de forma sucesiva a la red, en nuestro caso es muy sencillo. Primero debemos transformar la entrada que será una matriz de 28x28 en un vector de 784, para ello utilizaremos la función view, equivalente a reshape en numpy (el -1 se hace para que la primera dimensión, el tamaño de batch no sea alterada). Después aplicamos un función de activación a la salida de la capa oculta, en este caso hemos optado por la función rectified linear unit. Por último aplicamos la función softmax al tensor devuelto por la capa de salida, el parámetro dim indica en que dimensión se efectuará la función. Por último calculamos la función de pérdida, no es una práctica recomendable añadir el cálculo de dicha función al método forward ya que en el caso de únicamente querer predecir sería un cálculo absurdo pero en este caso lo haremos para que nos sea más sencillo entrenar la red. El código de dicha clase será el siguiente:

class MLP(nn.Module): def __init__(self): super(MLP, self).__init__() self.fc1 = nn.Linear(28*28, 256)#capa oculta self.fc2 = nn.Linear(256, 10)#capa de salida self.loss_criterion = nn.CrossEntropyLoss()#Función de pérdida def forward(self, x, target): x = x.view(-1, 28*28)#transforma las imágenes de tamaño (n, 28, 28) a (n, 784) x = F.relu(self.fc1(x))#Función de activación relu en la salida de la capa oculta x = F.softmax(self.fc2(x), dim=1)#Función de activación softmax en la salida de la capa oculta loss = self.loss_criterion(x, target)#Calculo de la función de pérdida return x, loss

Ahora que tenemos el modelo de red neuronal creamos un objeto de dicha clase y el optimizador que vamos a utilizar para ajustar los pesos de la red. En nuestro caso será el descenso por gradiente estocástico el optimizador que utilizaremos con un ratio de aprendizaje de 0.1 y un momento de 0.9.

model = MLP() optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.9)

Entrenamiento del modelo

Ahora vamos a desarrollar una función para poder entrenar la red, ya que hay que hacer el forward tanto para entrenamiento como para test vamos a utilizar la misma función. Dicha función recibirá el modelo, el dataloader, el optimizador y si estamos entrenando o no. Debemos tener un bucle que itere sobre cada batch del dataloader, en cada iteración el dataloader devuelve una tupla que contiene los datos de entrada y la salida esperada. Para cada tupla debemos poner a cero los gradientes de los pesos (si estamos entrenando). Después convertimos la entrada y el objetivo a un objeto llamado Variable dentro de autograd, esto hace que el objeto sea capaz de almacenar los gradientes y propagarlos. Una vez hecho esto podemos realizar el forward sobre la red y obtendremos la predicción ( score) y el valor que devuelve la función de pérdida ( loss). Con el valor de la predicción calculamos la clase predicha simplemente eligiendo la que obtenga el mayor valor y después comprobamos cuantas han sido correctas. En el caso de estar entrenando devemos propagar los gradientes con el algoritmo backward y después utilizar el optimizador para actualizar los pesos de la red. La función que acabamos de describir sería la siguiente:

def evaluate(model, dataset_loader, optimizer, train=False): correct_cnt, ave_loss = 0, 0#Contador de aciertos y acumulador de la función de pérdida count = 0#Contador de muestras for batch_idx, (x, target) in enumerate(dataset_loader): count += len(x)#sumamos el tamaño de batch, esto es porque n_batches*tamaño_batch != n_muestras if train: optimizer.zero_grad()#iniciamos a 0 los valores de los gradiente x, target = Variable(x), Variable(target)#Convertimos el tensor a variable del modulo autograd score, loss = model(x, target)#realizamos el forward _, pred_label = torch.max(score.data, 1)#pasamos de one hot a número correct_cnt += (pred_label == target.data).sum()#calculamos el número de etiquetas correctas ave_loss += loss.data[0]#sumamos el resultado de la función de pérdida para mostrar después if train: loss.backward()#Calcula los gradientes y los propaga optimizer.step()#adaptamos los pesos con los gradientes propagados accuracy = correct_cnt/count#Calculamos la precisión total ave_loss /= count#Calculamos la pérdida media print ('==>>>loss: {:.6f}, accuracy: {:.4f}'.format(ave_loss, accuracy))#Mostramos resultados

Entrenar nuestro modelo ahora es muy sencillo, por ejemplo, si queremos realizar el proceso de entrenamiento durante 10 epochs testeando en cada una simplemente ejecutamos el siguiente código:

for epoch in range(10): print("Epoch: {}".format(epoch)) print("Train") evaluate(model, train_loader, optimizer, train=True) print("Test") evaluate(model, test_loader, optimizer, train=False)

Guardar y cargar el modelo entrenado

Si hemos seguido todos los pasos, incluido fijar la semilla de PyTorch obtendremos una precisión de Test del 92.73 % expresado en tanto por uno.
Una vez entrenado el modelo podemos guardarlo y cargarlo fácilmente, para ello tenemos dos opciones, guardar el modelo entero o guardar solo los pesos. La primera opción es quizá la más sencilla:

torch.save(model, "nombre_del_modelo") model = torch.load("nombre_del_modelo")#Carga el modelo

Una vez cargado, el modelo es el mismo al que teníamos antes y podemos volver a entrenar o pasar el test. En el caso de querer guardar solo los pesos, la operación es igual de sencilla aunque a la hora de cargarlos debemos generar primero el modelo al igual que hemos hecho antes de entrenarlo y después cargar los pesos:

torch.save(model.state_dict(),"nombre_del_modelo") model = MLP()#Generamos un nuevo modelo model.load_state_dict(torch.load("nombre_del_modelo"))#Cargamos los pesos anteriores

Este método es el recomendado para realizar el guardado y así evitar incompatibilidades debidas a la serialización del objeto.

Modelo completo con integración en CUDA

Para hacer más sencilla la integración hemos diseñado la siguiente clase que permite realizar el entrenamiento y clasificación de una forma más sencilla y permite su integración con CUDA para el cálculo en GPU. De esta forma tendremos una clase base donde lo único que modificaremos será el __init__ y el forward de la clase NET para entrenar una topología diferente.

import torch import torch.nn as nn import torch.nn.functional as F from torch.autograd import Variable import torchvision.datasets as dset import torchvision.transforms as transforms import torch.optim as optim import time CUDA = torch.cuda.is_available() class NET(nn.Module): def __init__(self): super(NET, self).__init__() self.fc1 = nn.Linear(28*28, 256)#capa oculta self.fc2 = nn.Linear(256, 10)#capa de salida self.loss_criterion = nn.CrossEntropyLoss()#Función de pérdida def forward(self, x, target): x = x.view(-1, 28*28)#transforma las imágenes de tamaño (n, 28, 28) a (n, 784) x = F.relu(self.fc1(x))#Función de activación relu en la salida de la capa oculta x = F.softmax(self.fc2(x), dim=1)#Función de activación softmax en la salida de la capa oculta return x class NN(): def __init__(self): self.model = NET() if CUDA: self.model.cuda() self.optimizer = optim.SGD(self.model.parameters(), lr=0.01, momentum=0.9) def epoch_step(self, dataset_loader, train=False, verbose=2): correct_cnt, ave_loss = 0, 0#Contador de aciertos y acumulador de la función de pérdida count = 0#Contador de muestras for batch_idx, (x, target) in enumerate(dataset_loader): start_time_epch = time.time() count += len(x)#sumamos el tamaño de batch, esto es porque n_batches*tamaño_batch != n_muestras if train: self.optimizer.zero_grad()#iniciamos a 0 los valores de los gradiente x, target = Variable(x), Variable(target)#Convertimos el tensor a variable del modulo autograd if CUDA: x = x.cuda() target = target.cuda() score = self.model(x, target)#realizamos el forward loss = self.model.loss_criterion(score, target) _, pred_label = torch.max(score.data, 1)#pasamos de one hot a número correct_cnt_epch = (pred_label == target.data).sum()#calculamos el número de etiquetas correctas correct_cnt += correct_cnt_epch ave_loss += loss.data[0]#sumamos el resultado de la función de pérdida para mostrar después if train: loss.backward()#Calcula los gradientes y los propaga self.optimizer.step()#adaptamos los pesos con los gradientes propagados elapsed_time_epch = time.time() - start_time_epch if verbose == 1: elapsed_time_epch = time.strftime("%Hh,%Mm,%Ss", time.gmtime(elapsed_time_epch)) print ('\t\tbatch: {} loss: {:.6f}, accuracy: {:.4f}, time: {}'.format( batch_idx, loss.data[0], correct_cnt_epch/len(x), elapsed_time_epch)) accuracy = correct_cnt/count#Calculamos la precisión total ave_loss /= count#Calculamos la pérdida media return ave_loss, accuracy def train(self, epoch, train_loader, test_loader=None, verbose=2): for epoch in range(epoch): if verbose > 0: print("\n***Epoch {}***\n".format(epoch)) if verbose == 1: print("\tTraining:") start_time = time.time() ave_loss, accuracy = self.epoch_step(train_loader, train=True, verbose=verbose) elapsed_time = time.time() - start_time if verbose == 2: elapsed_time = time.strftime("%Hh,%Mm,%Ss", time.gmtime(elapsed_time)) print ('\tTraining loss: {:.6f}, accuracy: {:.4f}, time: {}'.format( ave_loss, accuracy, elapsed_time)) if test_loader != None: start_time = time.time() ave_loss, accuracy = self.epoch_step(test_loader, train=False, verbose=verbose) elapsed_time = time.time() - start_time if verbose == 2: elapsed_time = time.strftime("%Hh,%Mm,%Ss", time.gmtime(elapsed_time)) print ('\tTest loss: {:.6f}, accuracy: {:.4f}, time: {}'.format( ave_loss, accuracy, elapsed_time)) def evaluate(self, test_loader, verbose=2): print("\n***Evaluate***\n") start_time = time.time() ave_loss, accuracy = self.epoch_step(test_loader, train=False, verbose=verbose) elapsed_time = time.time() - start_time if verbose == 2: elapsed_time = time.strftime("%Hh,%Mm,%Ss", time.gmtime(elapsed_time)) print ('\tloss: {:.6f}, accuracy: {:.4f}, time: {}'.format( ave_loss, accuracy, elapsed_time)) def save_weights(self, path): torch.save(self.model.state_dict(), path) def load_weights(self, path): self.model.load_state_dict(torch.load(path)) if __name__ == "__main__": torch.manual_seed(123) #fijarmos la semilla epochs = 10 trans = transforms.Compose([transforms.ToTensor()]) #Transformador para el dataset root="./data" train_set = dset.MNIST(root=root, train=True, transform=trans, download=True) test_set = dset.MNIST(root=root, train=False, transform=trans) batch_size = 128 train_loader = torch.utils.data.DataLoader( dataset=train_set, batch_size=batch_size, shuffle=True, pin_memory=CUDA) test_loader = torch.utils.data.DataLoader( dataset=test_set, batch_size=batch_size, shuffle=False, pin_memory=CUDA) print ('Trainning batch number: {}'.format(len(train_loader))) print ('Testing batch number: {}'.format(len(test_loader))) net = NN() net.train(epochs, train_loader, test_loader=test_loader, verbose=2) net.evaluate(test_loader, verbose=2) net.save_weights("./weights")

En el inicio tenemos una variable booleana llamada CUDA, esta se pondrá a True si disponemos de una GPU NVIDIA en nuestro ordenador. Pese a que tengamos una GPU NVIDIA, no todas puede ejecutar PyTorch ya que necesita que la GPU tenga una capacidad de computo 3.0 por lo menos, podemos consultar una tabla que nos indica que capacidad tiene la nuestra aquí. En el caso de que la gráfica tenga una potencia inferior, posiblemente recibiremos varios errores donde uno de ellos será el siguiente:

torch.save(model.state_dict(),"nombre_del_modelo") model = MLP()#Generamos un nuevo modelo model.load_state_dict(torch.load("nombre_del_modelo"))#Cargamos los pesos anteriores

Para solucionar el error debemos ejecutar el código en la CPU, para ello simplemente ponemos la variable CUDA a False y ejecutará todos los cálculos en nuestra CPU.

# data[].attributes.attachment_link.title
Como crear tu primera red neuronal con PyTorch - Cleverpy

# data[].attributes.attachment_link.description
En el artículo PyTorch, ¿Qué es y cómo se instala? dimos a conocer el paquete PyTorch, un paquete de computación matemática diseñado para trabajar con tensores, está centrado para su utilización en el campo del machine learning, más en concreto en el desarrollo de redes neuronales.