# data[].attributes.headline
Machine Learning — Andrew Ng, Stanford University [FULL COURSE]

# data[].attributes.subject
Machine Learning — Andrew Ng, Stanford University [FULL COURSE]

# data[].attributes.body
Machine Learning — Andrew Ng, Stanford University [FULL COURSE]

# data[].attributes.attachment_link.title
Machine Learning - Andrew Ng, Stanford University [FULL COURSE]

# data[].attributes.attachment_link.description
About this course ------------------------------ Machine learning is the science of getting computers to act without being explicitly programmed. In the past...

# data[].attributes.headline
Constructing a Decision Tree for given dataset by Calculating Information Gain

# data[].attributes.subject
Constructing a Decision Tree for given dataset by Calculating Information Gain

# data[].attributes.attachment_link.title
Decision Tree Worked Example - print version for class exercise

# data[].attributes.body
1. Define Learning. Explain with an example.

# data[].attributes.body
2. What is the difference between a learning algorithm and a conventional algorithm run on a computer? Explain with an example.

# data[].attributes.headline
Abhishek Vijayvargia: Machine Learning with Python- an approach to applied ML, BPB publications

# data[].attributes.subject
Abhishek Vijayvargia: Machine Learning with Python- an approach to applied ML, BPB publications

# data[].attributes.attachment_link.title
Abhishek Vijayvargia - Machine Learning with Python-BPB Publications (2018).pdf

# data[].attributes.headline
Sunila Gollapudi: Practical Machine Learning, Packt Publishing Ltd

# data[].attributes.subject
Sunila Gollapudi: Practical Machine Learning, Packt Publishing Ltd

# data[].attributes.attachment_link.title
Gollapudi Sunila - Practical Machine Learning-Packt Publishing (2016).pdf

# data[].attributes.headline
1. John Paul Mueller - Machine Learning for Dummies

# data[].attributes.subject
1. John Paul Mueller - Machine Learning for Dummies

# data[].attributes.attachment_link.title
John Paul Mueller - Machine Learning For Dummies-For Dummies (2021).pdf

# data[].attributes.headline
2. Ethem Alpaydin - Introduction to Machine Learning, 3rd Edition, 2014

# data[].attributes.subject
2. Ethem Alpaydin - Introduction to Machine Learning, 3rd Edition, 2014

# data[].attributes.attachment_link.title
Ethem Alpaydin - Introduction to Machine Learning-The MIT Press (2014).pdf

# data[].attributes.attachment_link.title
Live Day 1- Introduction To Machine Learning Algorithms For Data Science

# data[].attributes.headline
Krish Naik - Complete Machine Learning In 6 Hours

# data[].attributes.subject
Krish Naik - Complete Machine Learning In 6 Hours

# data[].attributes.body
Download the video if possible

# data[].attributes.attachment_link.title
Complete Machine Learning In 6 Hours| Krish Naik

# data[].attributes.body
3. Elaborate machine learning (ML)

# data[].attributes.body
4. What are the different types of machine learning? Explain.

# data[].attributes.body
5. Explain different machine learning problem categories.

# data[].attributes.body
6. Explain machine learning process life cycle and architecture.

# data[].attributes.attachment_link.readable_text
Machine Learning tutorial provides basic and advanced concepts of machine learning. Our machine learning tutorial is designed for students and working professionals.

Machine learning is a growing technology which enables computers to learn automatically from past data. Machine learning uses various algorithms for building mathematical models and making predictions using historical data or information. Currently, it is being used for various tasks such as image recognition, speech recognition, email filtering, Facebook auto-tagging, recommender system, and many more.

This machine learning tutorial gives you an introduction to machine learning along with the wide range of machine learning techniques such as Supervised, Unsupervised, and Reinforcement learning. You will learn about regression and classification models, clustering methods, hidden Markov models, and various sequential models.

What is Machine Learning

In the real world, we are surrounded by humans who can learn everything from their experiences with their learning capability, and we have computers or machines which work on our instructions. But can a machine also learn from experiences or past data like a human does? So here comes the role of Machine Learning.

Machine Learning is said as a subset of artificial intelligence that is mainly concerned with the development of algorithms which allow a computer to learn from the data and past experiences on their own. The term machine learning was first introduced by Arthur Samuel in 1959. We can define it in a summarized way as:

Machine learning enables a machine to automatically learn from data, improve performance from experiences, and predict things without being explicitly programmed.

With the help of sample historical data, which is known as training data, machine learning algorithms build a mathematical model that helps in making predictions or decisions without being explicitly programmed. Machine learning brings computer science and statistics together for creating predictive models. Machine learning constructs or uses the algorithms that learn from historical data. The more we will provide the information, the higher will be the performance.
A machine has the ability to learn if it can improve its performance by gaining more data.
How does Machine Learning work

A Machine Learning system learns from historical data, builds the prediction models, and whenever it receives new data, predicts the output for it. The accuracy of predicted output depends upon the amount of data, as the huge amount of data helps to build a better model which predicts the output more accurately.

Suppose we have a complex problem, where we need to perform some predictions, so instead of writing a code for it, we just need to feed the data to generic algorithms, and with the help of these algorithms, machine builds the logic as per the data and predict the output. Machine learning has changed our way of thinking about the problem. The below block diagram explains the working of Machine Learning algorithm:

Features of Machine Learning:

Machine learning uses data to detect various patterns in a given dataset.

It can learn from past data and improve automatically.

It is a data-driven technology.

Machine learning is much similar to data mining as it also deals with the huge amount of the data.

Need for Machine Learning

The need for machine learning is increasing day by day. The reason behind the need for machine learning is that it is capable of doing tasks that are too complex for a person to implement directly. As a human, we have some limitations as we cannot access the huge amount of data manually, so for this, we need some computer systems and here comes the machine learning to make things easy for us.

We can train machine learning algorithms by providing them the huge amount of data and let them explore the data, construct the models, and predict the required output automatically. The performance of the machine learning algorithm depends on the amount of data, and it can be determined by the cost function. With the help of machine learning, we can save both time and money.

The importance of machine learning can be easily understood by its uses cases, Currently, machine learning is used in self-driving cars, cyber fraud detection, face recognition, and friend suggestion by Facebook, etc. Various top companies such as Netflix and Amazon have build machine learning models that are using a vast amount of data to analyze the user interest and recommend product accordingly.
Following are some key points which show the importance of Machine Learning:

Rapid increment in the production of data

Solving complex problems, which are difficult for a human

Decision making in various sector including finance

Finding hidden patterns and extracting useful information from data.

Classification of Machine Learning

At a broad level, machine learning can be classified into three types:

1. Supervised learning

2. Unsupervised learning

3. Reinforcement learning

1) Supervised Learning

Supervised learning is a type of machine learning method in which we provide sample labeled data to the machine learning system in order to train it, and on that basis, it predicts the output.

The system creates a model using labeled data to understand the datasets and learn about each data, once the training and processing are done then we test the model by providing a sample data to check whether it is predicting the exact output or not.

The goal of supervised learning is to map input data with the output data. The supervised learning is based on supervision, and it is the same as when a student learns things in the supervision of the teacher. The example of supervised learning is spam filtering.

Supervised learning can be grouped further in two categories of algorithms:

Classification

Regression

2) Unsupervised Learning

Unsupervised learning is a learning method in which a machine learns without any supervision.

The training is provided to the machine with the set of data that has not been labeled, classified, or categorized, and the algorithm needs to act on that data without any supervision. The goal of unsupervised learning is to restructure the input data into new features or a group of objects with similar patterns.

In unsupervised learning, we don't have a predetermined result. The machine tries to find useful insights from the huge amount of data. It can be further classifieds into two categories of algorithms:

3) Reinforcement Learning

Reinforcement learning is a feedback-based learning method, in which a learning agent gets a reward for each right action and gets a penalty for each wrong action. The agent learns automatically with these feedbacks and improves its performance. In reinforcement learning, the agent interacts with the environment and explores it. The goal of an agent is to get the most reward points, and hence, it improves its performance.

The robotic dog, which automatically learns the movement of his arms, is an example of Reinforcement learning.

Note: We will learn about the above types of machine learning in detail in later chapters.

History of Machine Learning

Before some years (about 40-50 years), machine learning was science fiction, but today it is the part of our daily life. Machine learning is making our day to day life easy from self-driving cars to Amazon virtual assistant "Alexa". However, the idea behind machine learning is so old and has a long history. Below some milestones are given which have occurred in the history of machine learning:

The early history of Machine Learning (Pre-1940):

1834: In 1834, Charles Babbage, the father of the computer, conceived a device that could be programmed with punch cards. However, the machine was never built, but all modern computers rely on its logical structure.

1936: In 1936, Alan Turing gave a theory that how a machine can determine and execute a set of instructions.

The era of stored program computers:

1940: In 1940, the first manually operated computer, "ENIAC" was invented, which was the first electronic general-purpose computer. After that stored program computer such as EDSAC in 1949 and EDVAC in 1951 were invented.

1943: In 1943, a human neural network was modeled with an electrical circuit. In 1950, the scientists started applying their idea to work and analyzed how human neurons might work.

Computer machinery and intelligence:

1950: In 1950, Alan Turing published a seminal paper, "Computer Machinery and Intelligence," on the topic of artificial intelligence. In his paper, he asked, "Can machines think?"

Machine intelligence in Games:

1952: Arthur Samuel, who was the pioneer of machine learning, created a program that helped an IBM computer to play a checkers game. It performed better more it played.

1959: In 1959, the term "Machine Learning" was first coined by Arthur Samuel.

The first "AI" winter:

The duration of 1974 to 1980 was the tough time for AI and ML researchers, and this duration was called as AI winter.

In this duration, failure of machine translation occurred, and people had reduced their interest from AI, which led to reduced funding by the government to the researches.

Machine Learning from theory to reality

1959: In 1959, the first neural network was applied to a real-world problem to remove echoes over phone lines using an adaptive filter.

1985: In 1985, Terry Sejnowski and Charles Rosenberg invented a neural network NETtalk, which was able to teach itself how to correctly pronounce 20,000 words in one week.

1997: The IBM's Deep blue intelligent computer won the chess game against the chess expert Garry Kasparov, and it became the first computer which had beaten a human chess expert.

Machine Learning at 21st century

2006: In the year 2006, computer scientist Geoffrey Hinton has given a new name to neural net research as "deep learning," and nowadays, it has become one of the most trending technologies.

2012: In 2012, Google created a deep neural network which learned to recognize the image of humans and cats in YouTube videos.

2014: In 2014, the Chabot "Eugen Goostman" cleared the Turing Test. It was the first Chabot who convinced the 33% of human judges that it was not a machine.

2014: DeepFace was a deep neural network created by Facebook, and they claimed that it could recognize a person with the same precision as a human can do.

2016: AlphaGo beat the world's number second player Lee sedol at Go game. In 2017 it beat the number one player of this game Ke Jie.

2017: In 2017, the Alphabet's Jigsaw team built an intelligent system that was able to learn the online trolling. It used to read millions of comments of different websites to learn to stop online trolling.

Machine Learning at present:

Now machine learning has got a great advancement in its research, and it is present everywhere around us, such as self-driving cars, Amazon Alexa, Catboats, recommender system, and many more. It includes Supervised, unsupervised, and reinforcement learning with clustering, classification, decision tree, SVM algorithms, etc.

Modern machine learning models can be used for making various predictions, including weather prediction, disease prediction, stock market analysis, etc.

Prerequisites

Before learning machine learning, you must have the basic knowledge of followings so that you can easily understand the concepts of machine learning:

Fundamental knowledge of probability and linear algebra.

The ability to code in any computer language, especially in Python language.

Knowledge of Calculus, especially derivatives of single variable and multivariate functions.

Audience

Our Machine learning tutorial is designed to help beginner and professionals.

Problems

We assure you that you will not find any difficulty while learning our Machine learning tutorial. But if there is any mistake in this tutorial, kindly post the problem or error in the contact form so that we can improve it.

# data[].attributes.attachment_link.keywords[]
machine learning vs artificial intelligence

# data[].attributes.attachment_link.title
Machine Learning: What It is, Tutorial, Definition, Types - Javatpoint

# data[].attributes.attachment_link.description
Machine Learning: What It is, Tutorial, Definition, Types with Machine Learning, Machine Learning Tutorial, Machine Learning Introduction, What is Machine Learning, Data Machine Learning, Applications of Machine Learning, Machine Learning vs Artificial Intelligence, dimensionality reduction, deep learning, etc.

# data[].attributes.body
https://www.simplilearn.com/tutorials/machine-learning-tutorial/machine-learning-steps#:~:text=Machine%20learning%20is%20the%20process,that%20data%20to%20learn%20more

# data[].attributes.attachment_link.readable_text
Machine Learning is a fantastic new branch of science that is slowly taking over day-to-day life. From targeted ads to even cancer cell recognition, machine learning is everywhere. The high-level tasks performed by simple code blocks raise the question, "How is machine learning done?".

In this tutorial titled 'The Complete Guide to Understanding Machine Learning Steps', you will go through the steps involved in making a machine learning model.

What Is Machine Learning?

Machine learning is the process of making systems that learn and improve by themselves, by being specifically programmed.

The ultimate goal of machine learning is to design algorithms that automatically help a system gather data and use that data to learn more. Systems are expected to look for patterns in the data collected and use them to make vital decisions for themselves.

In general, machine learning is getting systems to think and act like humans, show human-like intelligence, and give them a brain. In the real world, there are existing machine learning models capable of tasks like :

Separating spam from actual emails, as seen in Gmail

Correcting grammar and spelling mistakes, as seen in autocorrect

Thanks to machine learning, the world has also seen design systems capable of exhibiting uncanny human-like thinking, which performs tasks like:

Object and image recognition

Detecting fake news

Understanding written or spoken words

Bots on websites that interact with humans, like humans

Self-driven cars

Figure 1: Machine learning

Machine Learning Steps

The task of imparting intelligence to machines seems daunting and impossible. But it is actually really easy. It can be broken down into 7 major steps :

1. Collecting Data:

As you know, machines initially learn from the data that you give them. It is of the utmost importance to collect reliable data so that your machine learning model can find the correct patterns. The quality of the data that you feed to the machine will determine how accurate your model is. If you have incorrect or outdated data, you will have wrong outcomes or predictions which are not relevant.

Make sure you use data from a reliable source, as it will directly affect the outcome of your model. Good data is relevant, contains very few missing and repeated values, and has a good representation of the various subcategories/classes present.

Figure 2: Collecting Data

2. Preparing the Data:

After you have your data, you have to prepare it. You can do this by :

Putting together all the data you have and randomizing it. This helps make sure that data is evenly distributed, and the ordering does not affect the learning process.

Cleaning the data to remove unwanted data, missing values, rows, and columns, duplicate values, data type conversion, etc. You might even have to restructure the dataset and change the rows and columns or index of rows and columns.

Visualize the data to understand how it is structured and understand the relationship between various variables and classes present.

Splitting the cleaned data into two sets - a training set and a testing set. The training set is the set your model learns from. A testing set is used to check the accuracy of your model after training.

Figure 3: Cleaning and Visualizing Data

3. Choosing a Model:

A machine learning model determines the output you get after running a machine learning algorithm on the collected data. It is important to choose a model which is relevant to the task at hand. Over the years, scientists and engineers developed various models suited for different tasks like speech recognition, image recognition, prediction, etc. Apart from this, you also have to see if your model is suited for numerical or categorical data and choose accordingly.

Figure 4: Choosing a model

4. Training the Model:

Training is the most important step in machine learning. In training, you pass the prepared data to your machine learning model to find patterns and make predictions. It results in the model learning from the data so that it can accomplish the task set. Over time, with training, the model gets better at predicting.

Figure 5: Training a model

5. Evaluating the Model:

After training your model, you have to check to see how it's performing. This is done by testing the performance of the model on previously unseen data. The unseen data used is the testing set that you split our data into earlier. If testing was done on the same data which is used for training, you will not get an accurate measure, as the model is already used to the data, and finds the same patterns in it, as it previously did. This will give you disproportionately high accuracy.

When used on testing data, you get an accurate measure of how your model will perform and its speed.

Figure 6: Evaluating a model

6. Parameter Tuning:

Once you have created and evaluated your model, see if its accuracy can be improved in any way. This is done by tuning the parameters present in your model. Parameters are the variables in the model that the programmer generally decides. At a particular value of your parameter, the accuracy will be the maximum. Parameter tuning refers to finding these values.

Figure 7: Parameter Tuning

7. Making Predictions

In the end, you can use your model on unseen data to make predictions accurately.

How to Implement Machine Learning Steps in Python?

You will now see how to implement a machine learning model using Python.

In this example, data collected is from an insurance company, which tells you the variables that come into play when an insurance amount is set. Using this, you will have to predict the insurance amount for a person. This data was collected from Kaggle.com, which has many reliable datasets.

You need to start by importing any necessary modules, as shown.

Figure 8: Importing necessary modules

Following this, you will import the data.

Figure 9: Importing data

Figure 10: Insurance dataset

Now, clean your data by removing duplicate values, and transforming columns into numerical values to make them easier to work with.

Figure 11: Cleaning Data

The final dataset becomes as shown.

Figure 12: Cleaned dataset

Now, split your dataset into training and testing sets.

Figure 13: Splitting the dataset

As you need to predict a numeral value based on some parameters, you will have to use Linear Regression. The model needs to learn on your training set. This is done by using the '.fit' command.

Figure 14: Choosing and training your model

Now, predict your testing dataset and find how accurate your predictions are.

Figure 15: Predicting using your model

1.0 is the highest level of accuracy you can get. Now, get your parameters.

Figure 16: Model Parameters

The above picture shows the hyperparameters which affect the various variables in your dataset.

Stay ahead of the tech-game with our PG Program in AI and Machine Learning in partnership with Purdue and in collaboration with IBM. Explore more!

Conclusion

In this tutorial titled 'The Complete Guide to Understanding Machine Learning Steps', you took a look at machine learning and the steps involved in creating a machine learning model. You finally implemented these steps using Python.

We hope this article clearly explained the process of creating a machine learning model. To learn more about machine learning and how to make machine learning models, check out Simplilearn's AI and Machine Learning Course If you have any questions or doubts, mention them in this article's comments section, and we'll have our experts answer them for you at the earliest.

Happy learning!

# data[].attributes.attachment_link.title
The Complete Guide to Machine Learning Steps

# data[].attributes.attachment_link.description
Machine Learning is a fantastic new branch of science that is slowly taking over day-to-day life. From targeted ads to even cancer cell recognition, machine learning is everywhere. The high-level tasks performed by simple code blocks raise the question, "How is machine learning done?".

# data[].attributes.body
What is training and it's algorithm?

# data[].attributes.body
Thank you for being such a gentleman!
We love you sir!❤️❤️

# data[].attributes.body
Thank you sir for making so much efforts.

# data[].attributes.body
Deadline: Saturday  27 Sept. 2022, 23:59hrs

# data[].attributes.body
Andrew Ng, New Course, (2022), Coursera - Supervised Machine Learning.

# data[].attributes.attachment_link.readable_text
About this Course

In the first course of the Machine Learning Specialization, you will:

* Build machine learning models in Python using popular machine learning libraries NumPy and scikit-learn. * Build and train supervised machine learning models for prediction and binary classification tasks, including linear regression and logistic regression The Machine Learning Specialization is a foundational online program created in collaboration between DeepLearning.AI and Stanford Online. In this beginner-friendly program, you will learn the fundamentals of machine learning and how to use these techniques to build real-world AI applications. This Specialization is taught by Andrew Ng, an AI visionary who has led critical research at Stanford University and groundbreaking work at Google Brain, Baidu, and Landing.AI to advance the AI field. This 3-course Specialization is an updated and expanded version of Andrew's pioneering Machine Learning course, rated 4.9 out of 5 and taken by over 4.8 million learners since it launched in 2012. It provides a broad introduction to modern machine learning, including supervised learning (multiple linear regression, logistic regression, neural networks, and decision trees), unsupervised learning (clustering, dimensionality reduction, recommender systems), and some of the best practices used in Silicon Valley for artificial intelligence and machine learning innovation (evaluating and tuning models, taking a data-centric approach to improving performance, and more.) By the end of this Specialization, you will have mastered key concepts and gained the practical know-how to quickly and powerfully apply machine learning to challenging real-world problems. If you're looking to break into AI or build a career in machine learning, the new Machine Learning Specialization is the best place to start.

Reset deadlines in accordance to your schedule.

Approx. 33 hours to complete

What you will learn

Build machine learning models in Python using popular machine learning libraries NumPy & scikit-learn

Build & train supervised machine learning models for prediction & binary classification tasks, including linear regression & logistic regression

Skills you will gain

Regularization to Avoid Overfitting

Gradient Descent

Supervised Learning

Linear Regression

Logistic Regression for Classification

Reset deadlines in accordance to your schedule.

Approx. 33 hours to complete

Instructor rating 4.95/5 (1,471 Ratings)

Offered by

DeepLearning.AI

DeepLearning.AI is an education technology company that develops a global community of AI talent.

DeepLearning.AIs expert-led educational experiences provide AI practitioners and non-technical professionals with the necessary tools to go all the way from foundational basics to advanced application, empowering them to build an AI-powered future.

Stanford University

The Leland Stanford Junior University, commonly referred to as Stanford University or Stanford, is an American private research university located in Stanford, California on an 8,180-acre (3,310 ha) campus near Palo Alto, California, United States.

go to previous testimonial

go to next testimonial

Syllabus - What you will learn from this course

Content Rating 99% (31,834 ratings)

Week 1: Introduction to Machine Learning

Welcome to the Machine Learning Specialization! You're joining millions of others who have taken either this or the original course, which led to the founding of Coursera, and has helped millions of other learners, like you, take a look at the exciting world of machine learning!

20 videos (Total 147 min)

See All

Week 2: Regression with multiple input variables

This week, you'll extend linear regression to handle multiple input features. You'll also learn some methods for improving your model's training and performance, such as vectorization, feature scaling, feature engineering and polynomial regression. At the end of the week, you'll get to practice implementing linear regression in code.

Week 3: Classification

This week, you'll learn the other type of supervised learning, classification. You'll learn how to predict categories using the logistic regression model. You'll learn about the problem of overfitting, and how to handle this problem with a method called regularization. You'll get to practice implementing logistic regression with regularization at the end of this week!

11 videos (Total 98 min) , 1 reading , 5 quizzes

See All

Reviews

TOP REVIEWS FROM SUPERVISED MACHINE LEARNING: REGRESSION AND CLASSIFICATION

It is the Best Course for Supervised Machine Learning!

Andrew Ng Sir has been like always has such important & difficult concepts of Supervised ML with such ease and great examples, Just amazing!

The explanation is clear, and all of the source codes provided in each jupyter notebook show a clear visualisation of how well the model learns or fits into the data when a parameter changes.

Goes into the details of Logistic regression & Classification. The course is designed in a very simple and easy to understand format and Andrew sir is amazing all through out the sessions.

I really enjoyed taking this course. I absolutely loved the labs which were very helpful to test my understanding of the material I had just learned. Thank you to Coursera for this course!

View all reviews

About the Machine Learning Specialization

The Machine Learning Specialization is a foundational online program created in collaboration between DeepLearning.AI and Stanford Online. This beginner-friendly program will teach you the fundamentals of machine learning and how to use these techniques to build real-world AI applications.

This Specialization is taught by Andrew Ng, an AI visionary who has led critical research at Stanford University and groundbreaking work at Google Brain, Baidu, and Landing.AI to advance the AI field. This 3-course Specialization is an updated version of Andrew's pioneering Machine Learning course, rated 4.9 out of 5 and taken by over 4.8 million learners since it launched in 2012. It provides a broad introduction to modern machine learning, including supervised learning (multiple linear regression, logistic regression, neural networks, and decision trees), unsupervised learning (clustering, dimensionality reduction, recommender systems), and some of the best practices used in Silicon Valley for artificial intelligence and machine learning innovation (evaluating and tuning models, taking a data-centric approach to improving performance, and more.) By the end of this Specialization, you will have mastered key concepts and gained the practical know-how to quickly and powerfully apply machine learning to challenging real-world problems. If you're looking to break into AI or build a career in machine learning, the new Machine Learning Specialization is the best place to start.

When will I have access to the lectures and assignments?

What will I get if I subscribe to this Specialization?

Is financial aid available?

More questions? Visit the Learner Help Center.

# data[].attributes.attachment_link.title
Supervised Machine Learning: Regression and Classification

# data[].attributes.attachment_link.description
In the first course of the Machine Learning Specialization, you will: * Build machine learning models in Python using popular machine learning libraries NumPy and scikit-learn. * Build and train supervised machine learning models for prediction and binary classification tasks, including linear regression and logistic regression The Machine Learning Specialization is a foundational online program created in collaboration between DeepLearning.AI and Stanford Online.

# data[].attributes.headline
3. Neural Network & Deep Learning (web ref) 

# data[].attributes.subject
3. Neural Network & Deep Learning (web ref) 

# data[].attributes.body
Due date -
28/09/2022
23:59 PM

# data[].attributes.attachment_link.title
ISE 1 components 2 - Problem Solving (TYCSE-22-23)

# data[].attributes.headline
ML | Rainfall prediction using Linear regression - GeeksforGeeks

# data[].attributes.subject
ML | Rainfall prediction using Linear regression - GeeksforGeeks

# data[].attributes.attachment_link.readable_text
Prerequisites: Linear regression

Rainfall Prediction is the application of science and technology to predict the amount of rainfall over a region. It is important to exactly determine the rainfall for effective use of water resources, crop productivity and pre-planning of water structures.

In this article, we will use Linear Regression to predict the amount of rainfall. Linear Regression tells us how many inches of rainfall we can expect.

The dataset is a public weather dataset from Austin, Texas available on Kaggle. The dataset can be found here.

Data Cleaning:
Data comes in all forms, most of it being very messy and unstructured. They rarely come ready to use. Datasets, large and small, come with a variety of issues- invalid fields, missing and additional values, and values that are in forms different from the one we require. In order to bring it to workable or structured form, we need to "clean" our data, and make it ready to use. Some common cleaning includes parsing, converting to one-hot, removing unnecessary data, etc.

In our case, our data has some days where some factors weren't recorded. And the rainfall in cm was marked as T if there was trace precipitation. Our algorithm requires numbers, so we can't work with alphabets popping up in our data. so we need to clean the data before applying it on our model

Cleaning the data in Python:

Once the data is cleaned, it can be used as an input to our Linear regression model. Linear regression is a linear approach to form a relationship between a dependent variable and many independent explanatory variables. This is done by plotting a line that fits our scatter plot the best, ie, with the least errors. This gives value predictions, ie, how much, by substituting the independent values in the line equation.

We will use Scikit-learn's linear regression model to train our dataset. Once the model is trained, we can give our own inputs for the various columns such as temperature, dew point, pressure, etc. to predict the weather based on these attributes.
Output :

The precipitation in inches for the input is: [[1.33868402]]

The precipitation trend graph:

Precipitation vs selected attributes graph:

A day (in red) having precipitation of about 2 inches is tracked across multiple parameters (the same day is tracker across multiple features such as temperature, pressure, etc). The x-axis denotes the days and the y-axis denotes the magnitude of the feature such as temperature, pressure, etc. From the graph, it can be observed that rainfall can be expected to be high when the temperature is high and humidity is high.

# data[].attributes.attachment_link.title
ML | Rainfall prediction using Linear regression - GeeksforGeeks

# data[].attributes.attachment_link.description
Prerequisites: Linear regression Rainfall Prediction is the application of science and technology to predict the amount of rainfall over a region. It is important to exactly determine the rainfall for effective use of water resources, crop productivity and pre-planning of water structures. In this article, we will use Linear Regression to predict the amount of rainfall.

# data[].attributes.attachment_link.readable_text
Machine Learning

Looking for a fun challenge to grow your data science skills?

Here's a real-world problem for you.

Problem Statement

The weather has a significant impact on the agricultural industry and because of that, being able to predict it helps farmers in their day-to-day decisions such as how to plan efficiently, minimize costs and maximize yields.

A major agricultural company needs you to help them maximize growth efficiency, save resources and optimize their production.

To achieve these things, the company needs to have an accurate weather prediction algorithm that will improve their decision-making on typical farming activities such as planting and irrigating.

Using historical weather information from their region, can you predict what the weather will be like in the next few days?

You now have a clear goal.

The goal 🥅

Predict the next day's weather based on three labels.

Let's now look at the data

📂 📂traintest
 ├── region_A_test.csv
 ├── region_B_test.csv
 ├── region_C_test.csv
 ├── region_D_test.csv
 └── region_E_test.csv
 ├── region_A_train.csv
 ├── region_B_train.csv
 ├── region_C_train.csv
 ├── region_D_train.csv
 ├── region_E_train.csv
 ├── solution_format.csv
 └── solution_train.csv

The data has been conveniently split into train and test datasets.

In each train and test, you're given weather data which consists of anonymized locations named region A through region E, which are all neighboring regions.

Here's a look at the first five rows of region_A_train.csv

The first thing you should notice is that the date column isn't a date but was anonymized to be some random value.

There is a total of 10 features, which are composed of temperature, precipitation, wind speed, wind speed direction, and atmospheric pressure

Then, looking at solution_format.csv

We can utilize the date column to join it with the training data and build a model.

Now that you have an idea about the goal and some information about the data given to you, it's time to get your hands dirty.

Get the data by registering for this data science competition and follow along with this article!

Installing libraries and models

First, install the weapon of our choice:

Load Libraries

Next, we load up some essential libraries for visualizations and machine learning.

Load the data

Let's read in all the data we have.

It's time for the fun part, visualizing the data.

Join all the regions together

Since all the region data share a primary key date, we can connect them with in pandas and set the keys as the region names.

I don't want the regions as the index, so we can reset the index and then rename some columns to get the data in the right shape.

Distribution of target class

Let's first visualize our target class.

It appears we have in our hands an imbalanced class, as the N label is dominating the rest of the classes.
Why is this a problem?
The model will be biased towards classes with a larger amount of samples.

This happens because the classifier has more information on classes with more samples, so it learns how to predict those classes better while it remains weak on the smaller classes.

In our case, the label N will be predicted more than other classes.

Let's keep this in mind and move forward to more visualizations.

Plotting the features

We have ten features in each region.

It's time to cook up some plots.

Since all the regions are used to predict the next day's weather, let's see whether all the regions share similar patterns in the features and whether any outliers or anomalies exist.

From the plots, we see that the patterns in the data are very similar except for regions C, D, E for min.wind.speed and avg.wind.speed which are on the lower scale.

Now that we've explored the data a little, we check for missing values in our data.

Using my custom helper function, there seems to be a large amount of data missing for the min.atmos.pressure variable

Let's also use a heatmap to visualize the missing data for that column.

There are multiple ways we can deal with missing data.

Let's first look at the distribution of the column with missing values.

Since the distribution of the column is pretty normal, let's impute the missing data with the mean.

We then do the same for the test data (full code in the notebook)

Converting data types

Let's first add the labels to our data.

Then we take a look at the categorical columns for our dataset.

We'll have to convert the categorical features, including the target variable to a numerical format.

Let's use scikit-learn's Label Encoder to do that.

Here's an example of using LabelEncoder() on the label column

Creating new weather features

Since we're given some weather features, there are interesting new features we can engineer.

One example is the Beaufort scale, which is "an empirical measure that relates wind speed to observed conditions at sea or on land."

Below is a function to do our feature engineering.

We can easily apply the feature engineering steps on the train and test dataset.

Let's look at our train data so far.

It's time to pivot our data into the longer format, which means instead of the region being a column, each feature will have its own region, such as avg.temp_A, avg.temp_B, until avg.temp_E for the rest of the features.

This way the data will be in the right shape to build the model.

We can use the pivot_table function in pandas to achieve that.

The column names aren't ideal, so I wrote a function to fix that.

Here's what the train data looks like so far!

After we do the same to the test data, it's time to split the train data.

LightGBM likes it when the categorical features are given the categorical data type, so let's do that.

We can use train_test_split to split our data into the training and evaluation sets.

It's time to build the LightGBM model!

Let's create a base lgb.LGBMClassifier for a simple prediction

We can then fit the training data.

Then we call the predict function on the evaluation data

Let's see how a base LightGBM classifier did.

A 99% accuracy can be meaningless for an imbalanced dataset, so we need more suitable metrics like precision, recall, and a confusion matrix.

Confusion matrix

Let's create a confusion matrix for our model predictions.

First, we need to get the class names and the labels that the label encoder gave so our plot can show the label names.

We then plot a non-normalized and normalized confusion matrix.

As you can see from the shade of the plot, our model is predicting the label N much more than others.

With the values you see in the plot above, we can compute some metrics that tell us how well our model is doing.

Classification Report

A classification report measures the quality of predictions from a classification algorithm.

It answers the question of how many predictions are True and how many are False.

More specifically, it uses True Positives, False Positives, True Negatives, and False Negatives to compute the metrics of precision, recall, and f1-score

Let's break down the output.

Precision - What percent of your predictions were correct?
Recall - What percent of the positive cases did you catch?
F1-score - The weighted average of Precision and Recall
support - The number of occurrences of each given class

If you still have a hard time understanding precision and recall, read this great explanation.

Calculating the metrics

Taking class 0, which is the label H , let's calculate the precision, recall, and f1-score from the values in the confusion matrix

TP - True Positive | FP - False PositivePrecision = TP/(FP+TP) = 3/(3+4+2) = 3/9 = 0.33F1-score = 2 * (Recall * Precision) / (Recall + Precision)
FN - False Negative | TN - True Negative
Recall = TP/(TP+FN) = 3/(3+1+0) = 3/4 = 0.75
 = 2 * 0.33 * 0.75 / (0.33 + 0.75)
 = 0.495 / 1.08
 = 0.458

Macro or weighted?

You might notice that there are three values for the overall F1-score 0.64, 0.53, 0.65 . Which value should you focus on?

In an imbalanced dataset where all classes are equally important, macro average is a good choice as it treats all classes equally.

If however, you want to assign greater weight to classes with more samples in the data, then the weighted average is preferred.

One more metric you can use is Precision-Recall, which is a useful measure of the success of prediction when the classes are very imbalanced.

Let's also plot the feature importance to see which features matter more.

From the plot above, the wind speed features and precipitation are the key features that are good predictors.

The Beaufort scale feature we created seems to have very low importance, so it might be better to remove them.

Interestingly, min.atmos.pressure in region A is the most important, wheres min.atmos.pressure in other regions are among the lowest in importance.

When you're satisfied with your model, you can save it as a pickle file with joblib

We have a simple model built. It's time to predict it on the test data.

Let's have a peek at the final submission file.

The labels are still encoded as numeric values; let's bring the actual label names back.

Since we already have the dictionary of the mapping of label names to numeric values, i.e. 'H' : 0, we can reverse the dictionary above to map the numbers to the names

# data[].attributes.attachment_link.description
Looking for a fun challenge to grow your data science skills? Here's a real-world problem for you. The weather has a significant impact on the agricultural industry and because of that, being able to predict it helps farmers in their day-to-day decisions such as how to plan efficiently, minimize costs and maximize yields.

# data[].attributes.body
1. https://www.javatpoint.com/machine-learning-decision-tree-classification-algorithm

2. https://www.geeksforgeeks.org/gini-impurity-and-entropy-in-decision-tree-ml/#:~:text=As%20discussed%20above%20entropy%20helps,be%20calculated%20by%20this%20formula.

3.

# data[].attributes.headline
Tutorial - Calculate the Information Gains for each given dataset. 

# data[].attributes.subject
Tutorial - Calculate the Information Gains for each given dataset. 

# data[].attributes.body
(Datasets 3/11/2022)

# data[].attributes.headline
Construct a Decision Tree by Calculating Gini Index 

# data[].attributes.subject
Construct a Decision Tree by Calculating Gini Index 

# data[].attributes.attachment_link.title
Copy of Simple Regression in Matrix Form

# data[].attributes.headline
ML Unit 3 - Classification: Logistic Regression & Neural Network, Lectures 1 to 5 combined

# data[].attributes.subject
ML Unit 3 - Classification: Logistic Regression & Neural Network, Lectures 1 to 5 combined

# data[].attributes.attachment_link.title
ML Unit 3 - Classification - Logistic Regression & Neural Network, Lectures 1 to 5

# data[].attributes.headline
ML Unit 4 - Classification - Decision Trees and Naïve Bayes

# data[].attributes.subject
ML Unit 4 - Classification - Decision Trees and Naïve Bayes

# data[].attributes.attachment_link.title
ML Unit 4 - Classification - Decision Trees and Naïve Bayes - KBK

# data[].attributes.headline
ML Unit 6 - Introduction to Graphical Models

# data[].attributes.subject
ML Unit 6 - Introduction to Graphical Models

# data[].attributes.attachment_link.title
ML Unit 6 - Introduction to Graphical Models - KBK

# data[].attributes.body
A website to hone your skills in machine learning and data science

# data[].attributes.headline
To add matplotlib library from python library In Expriment No 2

# data[].attributes.subject
To add matplotlib library from python library In Expriment No 2

# data[].attributes.attachment_link.title
Unit 4 - Information Gain (Decision Tree)

# data[].attributes.attachment_link.title
Paper 1_kumar2018_stock market trend prediction

# data[].attributes.attachment_link.title
Paper 2_Mukharjee_Stock market prediction using deep learning algorithms

# data[].attributes.attachment_link.title
S3 batch Experiment tracker - Google Sheets

# data[].attributes.attachment_link.title
S5 Batch - Experiment tracker - Google Sheets

# data[].attributes.attachment_link.title
S4 Batch Experiment tracker - Google Sheets