# data[].attributes.body
Data Wrangling = Transformation of raw data to another form that is useful generating actionable insights

Data Cleaning =                To determine inaccurate, incomplete or unreasonable data and then the detected errors are corrected by replacing, modifying or deleting the dirty data

Data Integration =     Integrate data from multiple views

1. Physical - Copy data to warehouse

2. Virtual - Keep the data only at the sources

Data Reduction =         Obtain a reduced data set Dimensionality R - reduce number of random variables or attributes, Numerosity R- replace original data volume by alternative, Data compression - reduce size but preserve its representation

Data Transformation = Normalization :

1. Min-max - Change to a new min and max

2. Z-score - Shift of values according to data distribution

3. Decimal - Move to decimal places to keep max value < 1

Discretization : Divide the range of continuous attribute into intervals because some algorithm accept categorical

Binning - partition into equal size, smoothing by bin means, smoothing by bin boundaries

Entropy-based - find the best split to divide into 2 sets

Info (s1,s2) = |s1|/|s| x E(s1) + |s2| /  |s| x E(s2)
Gain(v,s2) = E(s) - info (s1,s2)
H bawah ni E

# data[].attributes.body
100%

Chapter 1, 4, 5, 7, 9, 11

67% (Twice)

Chapter 2, 6

33% (Once)

Chapter 8

0% (Never)

Chapter 3, 10, 12

Most Favorite Questions

5V's (1)

Responsibilities of Data Scientist (1)

Type of Questions (1)

Malas da. lol

Buat sendiri

Bapak ar

Pahal kau

Jahat nya kamu

k

# data[].attributes.body
Structured data:

Contains a defined data type, format and structure.

E.g: traditional RDBMS, CSV files

Semi-structured data:

Textual data files with a discernible pattern that enables parsing.

E.g: XML, JSON

Quasi-structured data:

Textual data with unpredictable data formats that can be formatted with effort, tools, and time.

E.g: Web Clickstream Data

Unstructured data:

Data that has no inherent structure.

E.g: Text, PDF, Images and Video

# data[].attributes.body
Increase of storage capacities

Increase of processing power

Availability of data

# data[].attributes.body
Volume

the vast amount of data generated each second (scale of data)

Velocity

the speed at which data is generated

Variety

the different types of data available (diversity)

Veracity

the trustworthiness of data

Value

the meaningfulness of data, creation of actionable insights, data monetization etc.

# data[].attributes.body
Academia
- Communication and People Skill
- Industry Skill
- Business Process Skill
+ Data Skill
+ Analytical Skill
+ Technical Skill

Industry
+ Communication and People Skill
+ Industry Skill
+ Business Process Skill
- Data Skill
- Analytical Skill
- Technical Skill

# data[].attributes.headline
Business Intelligence (BI) vs. Data Science (DS)

# data[].attributes.subject
Business Intelligence (BI) vs. Data Science (DS)

# data[].attributes.body
Typical Technique

BI - Standard & ad hoc reporting, dashboards, alerts, queries, details on demand.

DS - Optimization, predictive modeling, forecasting, statistical analysis.

Data Types

BI - Structured data, traditional sources, manageable datasets.

DS - Structured/unstructed data, many types of sources, very large datasets.

Common Questions

BI - What happened last quarter? Where is the problem?

DS - What if? What will happend next?

# data[].attributes.body
Facebook

analyses location information for easier to find friends to connect.

identitfy global migration patterns

determine where the different football team fan bases live

Target

predict which customers are pregnant, to focus baby-related marketing to them.

Tesco PLC

collected refrigerator-related data points to monitor it's performance, towards proactive maintenance to cut down on energy costs.

Macy's Inc

adjust pricing of it's items in near-real time by monitoring demand and inventory.

Siemens

leveraged on sensor-data analytics and predictive maintenance to reduce train failures.

Google Flu Trends

aggregates google search queries to predict outbreaks of flu.

Electoral Campaign (Obama)

data was mined to determine voters that need convincing, to choose donor-specific fundraising programs.

# data[].attributes.body
Data Collection

Data is obtained from a variety of sources.

Data Preprocessing

Detect and remove errors or amend inconsistencies in data.

Data Analysis

Exploration of the main characteristics of data.

Data Mining

Discovery of patterns/relationship within the data.

Data Visualization

Presentation of data in pictorial or graphical format

# data[].attributes.body
Ask an interesting question

Get the data

Explore the data

Model the data

Communicate and visualize the results.

# data[].attributes.body
Extract huge volumes of data from multiple internal and external sources.

Thoroughly clean & prune data to discard irrelevant information.

Invent new algorithm to solve problems.

Explore & examine data from a variety of angles to determine hidden weaknesses, trends or opportunities.

Prepare data for use in predictive and prescriptive modeling.

# data[].attributes.body
Validity of Assumptions

Making ad-hoc explanations of data patterns

Over-generalizing

Communication

Validation of models, data pipeline integrity

Using statistical tests correctly

Prototype -> Production transitions

# data[].attributes.body
1. Descriptive

To summarize a characteristic of a set of data.

E.g. What is the mean number of student procrastinating during study week?

2. Exploratory

Analyze the data to see if there are patterns, trends, or relationships between variables.

E.g. What is the relationship between student procrastination and academic performance?

3. Inferential

Restatement of the proposed hypothesis as a question and would be answered by analyzing a different set of data.

E.g. Propose the hypothesis that among students, sleeping more than 5 hours a day is associated with better academic performance, based on sample population of UiTM students. Re-examine on a different dataset such as MMU students.

4. Predictive

Ask what are the set of predictors/factors for a particular behavior.

E.g. Which group of student are high likely to perform better academically next semester?

5. Causal

Ask about whether changing one factor will change another factor.

E.g. Will an increase in PUBG Lite play time increase the tendency to achieve Chicken Dinner?

6. Mechanistic

How a factor affects the outcome.

How Marvel Studio decrease the DC fan base entirely?

# data[].attributes.body
1. The question should of interest to your audience.

2. The question has not already been answered.

3. The question should also stem from a plausible (valid correlations) framework.

4. The question, should also be answerable.

5. The question should be specific.

# data[].attributes.body
Completeness

All necessary data have been recorded. Data can be complete even if optional data is missing.

Timeliness

The data is kept up to date. It is about having the right information at the right time.

Consistency

The data across processes, organizations, sources are in sync with each other.

Validity

The same fields are used consistently for the same information capture.

Accuracy

The data was recorded correctly.

Uniqueness

Each record is distinct and unique.

# data[].attributes.body
Descriptive

What is happening?

Comprehensive, accurate  and live data.

Diagnostic

Why did it happen?

Ability to drill down to the root-cause.

Predictive

What is likely to happen?

Historical patterns being used to predict specific outcomes using algorithms.

Prescriptive

What should i do about it?

Recommended actions & strategies based on challenger testing outcomes.

# data[].attributes.body
Numerical Data

Discrete Data

Values are distinct and separated.

E.g. Number of staff (15,16)

Continuous Data

Values may take on any value between finite and infinite interval.

E.g. Height and weight (85.5, 89.9)

Categorical Data

Nominal Data

Values can be assigned a code in the form of a number.

E.g. Gender (Male = 0, Female = 1)

Ordinal Data

Value can be ranked or having a rating scale attached.

E.g. Very unsatisfied = 0, Unsatisfied = 1, Neutral = 3, Satisfied = 4, Very satisfied = 5

# data[].attributes.body
Tabular

Frequency Distributions

Relative Frequency Distributions

Graphical

Bar Chart or Histogram

Stem and Leaf Plot

Scatter Plot

# data[].attributes.body
Measures of Central Tendency

Mean

Median

Mode

Measures of Dispersion of Data

Range

Quantiles

Quartiles

Interquartile Range

Percentiles

Boxplots

Variance

Standard Deviation

# data[].attributes.body
Five-number summary

Minimum

Quartile Q1

Median

Quartile Q3

Maximum

Boxplot

# data[].attributes.body
1. Incomplete data

Non available value

Different criteria between time data collected when it is analysed

Human/hardware/software problems

Ignore the tuple
Fill in missing value manually
Use a global constant to fill
Use the attribute mean/median Predict using a learning algorithm

2. Noisy data

Faulty instruments

Data entry or computer errors

Data transmission

Binning methods
Outlier analysis
Regression
Combination of human and computer inspection

3. Inconsistent / redundant

Different data sources

Functional dependencies

Referential integrity violation

Naming conventions
Parsing text into fields
Different representations
Primary key violation
Formatting issues

# data[].attributes.body
Lower inner fence = Q1 - 1.5*IQR
Upper inner fence = Q3 + 1.5*IQR
Lower outer fence = Q1 - 3*IQR
Upper outer fence = Q3 + 3*IQR

Mild outlier

Beyond an inner fence on either side

Extreme outlier

Beyond an outer fence on either side

# data[].attributes.body
Low standard deviation

data tend to be very close to the mean

High standard deviation

data are spread out over a large range of values

# data[].attributes.body
Skewness is the measure of the asymmetry of the probability of a real-valued random variable about its mean.

# data[].attributes.body
Kurtosis is a measure of the "tailedness" of the probability distribution of a real-valued random variable.

High (Positive)Kurtosis

"Heavy" tails, or potential outliers.

Low (Negative) Kurtosis

"Light" tails, or less likely having outliers.

# data[].attributes.body
Positive Correlation

one variable increases with the other.

Negative Correlation

one variable decreases when the other increases.

Correlation close to 0

both variables have little influence over the other.

# data[].attributes.body
Artificial Intelligence

the theory/development of computer systems able to perform task that normally require human intelligence

E.g. Visual perception, speech recognition, decision-making, translation between languages.

Data Mining

the practice of examining large databases to discover patterns or new information.

Machine Learning

the training of model from data that generalizes a decision(prediction) against a performance measure.

# data[].attributes.body
Unsupervised Learning discover patterns/relationship from the data itself

Clustering

discover the inherent groupings in the data.

E.g. K-means clustering

Association

discover rules that describe large portions of your data.

E.g. Apriori algorithm for association rule mining

Supervised Learning      based on "training examples". Has the ability to reach an accurate conclusion when given new data.

Classification

predicts an output variable as a category.

E.g. Decision Trees, Support Vector Machine.

Regression

predicts an output variable as a real value.

E.g. Linear Regression

# data[].attributes.body
The machine trains itself continually using trial and error. It learns from past experience and capture the best possible knowledge to make accurate business decisions.

E.g. Markov Decision Process

# data[].attributes.body
calculates the distance between p and q, where each observation has n variables.

# data[].attributes.body
1. Randomly pick k number of points for each cluster (centroids)

2. Each data point is assigned to the cluster with the closest centroids.

3. Compute the centroid of each cluster based on existing cluster members, this is result in new centroids.

4. As we have new centroids, repeat step 2 and 3. Repeat this process until convergence occurs. I.e. centroids does not change.

Ez explanation

1. Randomly pick k number of centroids for each cluster.

2. Each data is group into nearest centroids.

3. Move the centroid to the mean of the group.

4. Repeat 2 and 3 until centroid does'nt change

Limitations

Predefined number of clusters

self explanatory

Distorted by outliers

can result in non-optimal grouping. (outlier influence the mean value)

No hierarchical organization

Applications

Marketing

discover distinct groups in their customer bases.

Insurance

identify groups of insurance policy holders with a high average claim cost.

City-planning

identify groups of houses according to their house type, value, and geographical location.

DNA sequences

cluster DNA information based on edit distance

# data[].attributes.body
Support

How popular an itemset is per all the transactions.

E.g. Cereal and milk appear together in 40% of the transactions i.e Support = 40%

Confidence

How likely item Y is purchased when item X is purchased (X->Y)

E.g. Cereal (X) appear in 50 transactions; 40 of the 50 also include milk (Y). Cereal(X) implies milk(Y) with 80% confidence.

Lift

Like confidence but while controlling how popular item Y is.

Lift > 1: Item Y is likely to be bought if item X is bought.

Lift < 1: Item Y is unlikely to be bought if item X is bought.

Lift = 1: Those two item are independent of each other. referto

E.g. Orange milk have 75% confidence, 30% support. Customers generally buy milk 90% of the time and orange 40% of the time.

Lift = 0.3/(0.4*0.9) = 0.83

# data[].attributes.body
Building

Data used contains values for the descriptor(X) and response(Y) variables.

Training set

used for building the model.

Test set

accessing the quality of the model

Applying

Once model has been built, a dataset with no output response variables can be fed into the model to obtain estimated response.

# data[].attributes.body
You want to show individual data values, and allow audience to compare between individual values.

Precise values are required.

The quantitative information to be communicated involves more than one unit measure.

# data[].attributes.body
You want to reveal relationships between multiple values.

General patterns are the key point you want to present, rather than the exact data values.

# data[].attributes.body
Description

A simple comparison of the categorical subdivisions of one or more measure in no particular order.

Methods

Bars only (horizontal or vertical)

# data[].attributes.body
Description

Multiple instances of one or more measures taken at equidistant points in time.

Methods

Lines to emphasize overall pattern

Bars to emphasize individual values

Points connected by lines to slightly emphasize individual values while still highlighting the overall pattern.

Always place time on the horizontal axis

# data[].attributes.body
Filtering = Based on some conditions

Transforming = Add new variables

Aggregating = Collapse many to one

Sorting = Change the order of values

# data[].attributes.body
1. Stepwise forward selection - Start empty, best attribute picked

2. Stepwise backward selection - Start full, eliminate the worst

3. Combination - Select best and remove worst

4. Decision Tree Induction

# data[].attributes.body
Description

Categorical subdivisions of a measure ordered by size (either descending or ascending)

Methods

Bars only (horizontal or vertical)

To highlight high values, sort in descending order.

To highlight low values, sort in ascending order.

# data[].attributes.body
Description

Categorical subdivisions of a measure compared to a reference measure, expressed as the differences between them.

Methods

Lines to emphasize the overall pattern only when displaying deviation and time-series relationships together.

Points connected by lines to slightly emphasize individual data points while also highlighting the overall pattern when displaying deviation and time-series relationships together.

Bars to emphasize individual values, but limit to vertical bars when a time-series relationship is included.

# data[].attributes.body
Description

Measures of individual categorical subdivisions as ratios to the whole.

Methods

Bars only (horizontal or vertical)

Use stacked bars only when you must display measures of the whole as well as the parts.

# data[].attributes.headline
Needs Of Integration = Heterogeneity among data sources

# data[].attributes.subject
Needs Of Integration = Heterogeneity among data sources

# data[].attributes.body
1. Source type H Different store systems

2. Communication H some have web interfaces some offer API

3. Schema H Different table structures

4. Data type H Different data type stored

5. Value H Different ways of same values

6. Semantic H Different sources different things

# data[].attributes.body
Description

Counts of something per categorical subdivisions (intervals) of quantitative range.

Methods

Vertical bars to emphasize individual values (histogram)

Lines to emphasize the overall pattern (frequency polygon)

# data[].attributes.body
1. DataWatch Monarch

2. Trifacta Wrangler

3. Openrefine

# data[].attributes.body
Description

Comparisons of two paired sets of measures to determine if one set goes up, the other set goes either up or down in a corresponding manner, and it so, how strongly.

Methods

Points and trend line in the form of a scatter plot.

Bars may be used, arranged as a paired bar graph or correlation bar graph, if scatter plots are unfamiliar.

# data[].attributes.body
R (statistics)

Python (pandas, numpy)

Scala (enable data streaming)

SQL (RDBMS)

Excel (data analysis)

Java

Matlab (academic/math)

SPSS (statistical packages)

WEKA (machine learning)

RapidMiner (DS software platform)

Hadoop (process extremely large data sets)

D3.js (data visualization)

Tableau (data visualization)

Apache Spark (machine learning)

SAS (analytics software)

# data[].attributes.body
High degree of organisation such as a relational database

# data[].attributes.body
Information that is difficult to organise using traditional mechanics

# data[].attributes.body
link

Python

Advantages

Improvement in availability of Python packages.

A good tool to implement algorithms for production use.

R

Advantages

Easier for beginner to do exploratory work.

Usable for basic data analysis without installation of packages

# data[].attributes.body
UD 60-80% per year
1 exabytes = 1000 petabytes = 1 million terabytes = 1 billion gigabytes

# data[].attributes.body
Relational database (RDBMS)

Table based database

Web, mobile, enterprise, data mart

# data[].attributes.body
Non-relational/distributed database

Document based, key-value pairs, graph database or wide-column stores

Gaming, social, iot, web

# data[].attributes.body
Basically Available = Guaranteed availability

Soft-state = The state of the system may change, without even a query because of node updates

Eventually consistent = The system will become consistent over time

# data[].attributes.body
Atomicity = All data and commands in a transaction succeed, or all fail and roll back

Consistency = All committed data must be consistent with all data rules including constraints, triggers, cascade, atomicity, isolation and durability

Isolation = Other operations cannot access data that has been modified during a transaction that has not yet completed

Durability = Once a transaction is committed, data will survive system failures, and can be reliably recovered after an unwanted deletion

# data[].attributes.body
Use the notion of databases separated into tables where each column represents a field and each row represents a record

MySQL

PostgreSQL

SQLite3

# data[].attributes.body
1NF : Eliminate groups of repeating data by creating a newtable for each group of related data which is identified by a primary key

2NF : If a set of values are the same for multiple records move them to a new table and link the two tables with a foreign key

3NF : Fields which do not depend on the primary key of a table must be removed and if necessary  be put into another table

# data[].attributes.body
Atomicity : Either all parts of a transaction must be completed or none

Consistency : The integrity of the database is preserved by all transactions. The database is not left in an invalid state after a transaction

Isolation : A transaction must be run isolated in order to guarantee that any inconsistency in the data involved does not affect other transactions

Durability : The changes made by a completed transaction must be preserved or in other words be durable

# data[].attributes.body
Represent data in collection of JSON docs

Records can have different fields (Dynamic Schema)

1. Key-value stores Redis Voldemort Dynamo

2. Column-oriented database Cassandra Hbase link

3. Document-based stores MongoDB link

4. Graph databases Neo4J InfiniteGraph link

# data[].attributes.body
1. Generally process data faster than relational DB

2. Faster because their data models are simpler

3. No schema required

# data[].attributes.body
Best used: For rapidly changing data with a foreseeable database size

Use cases: To store real-time stock prices, Real-time analytics. Leaderboards. Real-time communication

# data[].attributes.body
Advantage of Column-oriented = Some types of data lookups can become very fast, given that the desired data could be stored consecutively in a single row

Best used: When you need to store data so huge that it doesnt fit on server, but still want a friendly familiar interface to it

Use cases: Web analytics, to count hits by hour, by browser, by IP. Transaction logging. Data collection from huge sensor arrays

# data[].attributes.body
Load balancing, replication, indexing, querying and act as file system

Best used: If you need dynamic queries. If you need good performance on big DB

Use cases: For most things that you would do with MySQL or PostgreSQL but having predefined columns

# data[].attributes.body
Advantage of Graph-based = Potential in certain data mining and pattern recognition scenarios

Best used: For graph-style, rich or complex, interconnected data

Use cases: For searching routes in social relations, public data transport links, road maps, or network topologies

# data[].attributes.body
Python package rpy2 allows one to call R from within Python

Can translate Python objects into R, pass into R functions, and convert the R output back into Python objects.

Python -> R functions -> R output -> Python

# data[].attributes.body
After this thing, just for reading/understanding only. Never ask one. Just know the basic coding using pandas & numpy.

# data[].attributes.body
A fast and efficient multidimensional array object ndarray

Tools for integrating C, C++ and Fortran code for Python

More efficient way of storing and manipulating data (numerical data).

# data[].attributes.body
The primary object in pandas is the DataFrame, a two-dimensional tabular, column oriented data structure with both row and column tables.

*this sht have a lot of stuff to read

Pandas Cheat Sheet!

# data[].attributes.body
Python library for producing plots and other 2D data visualizations.

# data[].attributes.body
Question 1

Key-enablers (1)

5V's (1)

Challenges in DS (1)

Data structures (1)

Question 2

Types of questions (2)

Heterogeneity (4)

Data Quality Dimensions (2)

Errors/Inconsistencies (4)

Question 3

Python code (5)

Correlation (6)

Machine learning (7)

Question 4

SQL and NoSQL (11)

Python is the preferred language (5)

Appropriate Chart/Graph (9)

Chapter 1, 2, 4, 5, 6, 7, 9 ,11
Yg x keluar, 3,8,10,12

# data[].attributes.body
Question 1

Responsiblities of Data Scientist (1)

5V's (1)

Big Data Insights (1)

Characteristics of Big Data (1)

Question 2

Types of questions (2)

Outliers (6)

Data Quality Dimensions (2)

Errors/Inconsistencies (4)

Question 3

Python code (5)

Machine learning (7)

Correlation (6)

Question 4

SQL and NoSQL (11)

R is the preferred language (5)

Graph/Table (9)

Chapter 1, 2, 4, 5, 6, 7, 9, 11
Yg x keluar, 3,8,10,12

# data[].attributes.body
Question 1

BI vs DS (1)

Responsibilities of Data Scientist (1)

Question 2

Entropy-based (4)

Calculate Gain (4)

NoSQL (11)

MongoDB (11)

Python packages contains R (5)

Question 3

Training set and Test Set (8)

Association Rules Mining - Confidence, Lift (7)

Euclidean Distance (7)

Question 4

Python Code (5)

Graph/Table (9)

Chapter 1, 4, 5, 7, 8, 9, 11
Yg x keluar, 2, 3, 6, 10 ,12

# data[].attributes.headline
Traditional Database vs Modern Database

# data[].attributes.subject
Traditional Database vs Modern Database

# data[].attributes.body
- Data Sources
Both: MS SQL
T: Tables, Files
M: Social, Streaming

- Data Structure
Both: Structured data
T: ER-model, Multi-dimensional Model
M: Unstructured data, Machine data

- Data Access
T: SQL
M: Parallel processing, Distributed processing, In-memory processing

- Analytics
T: Reporting, Dashboard, Data Analysis
M: Regression, Clustering, Data Mining

# data[].attributes.body
Simpson Paradox states that the representation of the data will be misleading without considering the confounding factors.

# data[].attributes.body
- Classification accuracy
Sensitivity (Recall) = TP / P
Precision = TP / P'
F1-score
P = all actual positive values
P' = all predicted positive values
F1 = 2 x R x P / (R+P)

Regression metrics:
- MAE
- MSE
-

- Area under the ROC graph

# data[].attributes.body
to represent values on a map with different shade of colour to represent the value.

# data[].attributes.body
https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm