Introduction To Data Science YEEETT by Shahrin Amin

Data Structures

shahrinamin_my — 2019-03-07 08:29:43 UTC

Structured data:
- Contains a defined data type, format and structure.
- E.g: traditional RDBMS, CSV files
Semi-structured data:
- Textual data files with a discernible pattern that enables parsing.
- E.g: XML, JSON
Quasi-structured data:
- Textual data with unpredictable data formats that can be formatted with effort, tools, and time.
- E.g: Web Clickstream Data
Unstructured data:
- Data that has no inherent structure.
- E.g: Text, PDF, Images and Video

Key Enablers for the growth of Big Data

shahrinamin_my — 2019-03-07 08:42:10 UTC

Increase of storage capacities
Increase of processing power
Availability of data

Characterization of Big Data

shahrinamin_my — 2019-03-07 08:46:41 UTC

Volume
- the vast amount of data generated each second (scale of data)
Velocity
- the speed at which data is generated
Variety
- the different types of data available (diversity)
Veracity
- the trustworthiness of data
Value
- the meaningfulness of data, creation of actionable insights, data monetization etc.

Insights from Big Data

shahrinamin_my — 2019-03-07 08:57:25 UTC

Facebook
- analyses location information for easier to find friends to connect.
- identitfy global migration patterns
- determine where the different football team fan bases live
Target
- predict which customers are pregnant, to focus baby-related marketing to them.
Tesco PLC
- collected refrigerator-related data points to monitor it's performance, towards proactive maintenance to cut down on energy costs.
Macy's Inc
- adjust pricing of it's items in near-real time by monitoring demand and inventory.
Siemens
- leveraged on sensor-data analytics and predictive maintenance to reduce train failures.
Google Flu Trends
- aggregates google search queries to predict outbreaks of flu.
Electoral Campaign (Obama)
- data was mined to determine voters that need convincing, to choose donor-specific fundraising programs.

Business Intelligence (BI) vs. Data Science (DS)

shahrinamin_my — 2019-03-07 09:13:29 UTC

Typical Technique

BI - Standard & ad hoc reporting, dashboards, alerts, queries, details on demand.
DS - Optimization, predictive modeling, forecasting, statistical analysis.

Data Types

BI - Structured data, traditional sources, manageable datasets.
DS - Structured/unstructed data, many types of sources, very large datasets.

Common Questions

BI - What happened last quarter? Where is the problem?
DS - What if? What will happend next?

Data Science Pipelines

shahrinamin_my — 2019-03-07 09:30:40 UTC

Data Collection
- Data is obtained from a variety of sources.
Data Preprocessing
- Detect and remove errors or amend inconsistencies in data.
Data Analysis
- Exploration of the main characteristics of data.
Data Mining
- Discovery of patterns/relationship within the data.
Data Visualization
- Presentation of data in pictorial or graphical format

Data Science Process

shahrinamin_my — 2019-03-07 09:34:19 UTC

Ask an interesting question
Get the data
Explore the data
Model the data
Communicate and visualize the results.

Responsibilities of a Data Scientist

shahrinamin_my — 2019-03-07 09:34:54 UTC

Extract huge volumes of data from multiple internal and external sources.
Thoroughly clean & prune data to discard irrelevant information.
Invent new algorithm to solve problems.
Explore & examine data from a variety of angles to determine hidden weaknesses, trends or opportunities.
Prepare data for use in predictive and prescriptive modeling.

Challenges in Data Science

shahrinamin_my — 2019-03-07 09:39:04 UTC

Validity of Assumptions
Making ad-hoc explanations of data patterns
Over-generalizing
Communication
Validation of models, data pipeline integrity
Using statistical tests correctly
Prototype -> Production transitions

Basic Types of Questions

shahrinamin_my — 2019-03-07 09:44:01 UTC

Descriptive
- To summarize a characteristic of a set of data.
- E.g. What is the mean number of student procrastinating during study week?
Exploratory
- Analyze the data to see if there are patterns, trends, or relationships between variables.
- E.g. What is the relationship between student procrastination and academic performance?
Inferential
- Restatement of the proposed hypothesis as a question and would be answered by analyzing a different set of data.
- E.g. Propose the hypothesis that among students, sleeping more than 5 hours a day is associated with better academic performance, based on sample population of UiTM students. Re-examine on a different dataset such as MMU students.
Predictive
- Ask what are the set of predictors/factors for a particular behavior.
- E.g. Which group of student are high likely to perform better academically next semester?
Causal
- Ask about whether changing one factor will change another factor.
- E.g. Will an increase in PUBG Lite play time increase the tendency to achieve Chicken Dinner?
Mechanistic
- How a factor affects the outcome.
- How Marvel Studio decrease the DC fan base entirely?

Characteristics of Good Question

shahrinamin_my — 2019-03-07 10:07:02 UTC

The question should of interest to your audience.
The question has not already been answered.
The question should also stem from a plausible (valid correlations) framework.
The question, should also be answerable.
The question should be specific.

Dimensions of Data Quality

shahrinamin_my — 2019-03-07 10:14:05 UTC

Completeness
- All necessary data have been recorded. Data can be complete even if optional data is missing.
Timeliness
- The data is kept up to date. It is about having the right information at the right time.
Consistency
- The data across processes, organizations, sources are in sync with each other.
Validity
- The same fields are used consistently for the same information capture.
Accuracy
- The data was recorded correctly.
Uniqueness
- Each record is distinct and unique.

4 Types of Analytics

shahrinamin_my — 2019-03-08 08:19:36 UTC

Descriptive
- What is happening?
- Comprehensive, accurate and live data.
Diagnostic
- Why did it happen?
- Ability to drill down to the root-cause.
Predictive
- What is likely to happen?
- Historical patterns being used to predict specific outcomes using algorithms.
Prescriptive
- What should i do about it?
- Recommended actions & strategies based on challenger testing outcomes.

No Final Question

2019-03-08 08:20:58 UTC

Data Types

shahrinamin_my — 2019-03-08 08:45:20 UTC

Numerical Data
- Discrete Data
  - Values are distinct and separated.
  - E.g. Number of staff (15,16)
- Continuous Data
  - Values may take on any value between finite and infinite interval.
  - E.g. Height and weight (85.5, 89.9)
Categorical Data
- Nominal Data
  - Values can be assigned a code in the form of a number.
  - E.g. Gender (Male = 0, Female = 1)
- Ordinal Data
  - Value can be ranked or having a rating scale attached.
  - E.g. Very unsatisfied = 0, Unsatisfied = 1, Neutral = 3, Satisfied = 4, Very satisfied = 5

Numerical Data

shahrinamin_my — 2019-03-08 10:59:10 UTC

Describing Data

shahrinamin_my — 2019-03-08 11:00:25 UTC

Tabular
- Frequency Distributions
- Relative Frequency Distributions
Graphical
- Bar Chart or Histogram
- Stem and Leaf Plot
- Scatter Plot

Statistical Description

shahrinamin_my — 2019-03-08 11:05:06 UTC

Measures of Central Tendency
- Mean
- Median
- Mode
Measures of Dispersion of Data
- Range
- Quantiles
- Quartiles
- Interquartile Range
- Percentiles
- Boxplots
- Variance
- Standard Deviation

Five-number Summary, Boxplot

shahrinamin_my — 2019-03-08 11:12:31 UTC

Five-number summary
- Minimum
- Quartile Q1
- Median
- Quartile Q3
- Maximum
Boxplot

Outliers

shahrinamin_my — 2019-03-08 11:25:35 UTC

Lower inner fence = Q1 - 1.5*IQR
Upper inner fence = Q3 + 1.5*IQR
Lower outer fence = Q1 - 3*IQR
Upper outer fence = Q3 + 3*IQR

Mild outlier
- Beyond an inner fence on either side
Extreme outlier
- Beyond an outer fence on either side

Variance and Standard Deviation

shahrinamin_my — 2019-03-08 11:29:11 UTC

Low standard deviation
- data tend to be very close to the mean
High standard deviation
- data are spread out over a large range of values

Skewness

shahrinamin_my — 2019-03-08 12:27:23 UTC

Skewness is the measure of the asymmetry of the probability of a real-valued random variable about its mean.

Kurtosis

shahrinamin_my — 2019-03-08 12:41:48 UTC

Kurtosis is a measure of the "tailedness" of the probability distribution of a real-valued random variable.

High (Positive)Kurtosis
- "Heavy" tails, or potential outliers.
Low (Negative) Kurtosis
- "Light" tails, or less likely having outliers.

Correlation Analysis

shahrinamin_my — 2019-03-08 12:55:11 UTC

Positive Correlation
- one variable increases with the other.
Negative Correlation
- one variable decreases when the other increases.
Correlation close to 0
- both variables have little influence over the other.

AI vs DM vs ML

shahrinamin_my — 2019-03-08 16:28:16 UTC

Artificial Intelligence
- the theory/development of computer systems able to perform task that normally require human intelligence
- E.g. Visual perception, speech recognition, decision-making, translation between languages.
Data Mining
- the practice of examining large databases to discover patterns or new information.
Machine Learning
- the training of model from data that generalizes a decision(prediction) against a performance measure.

Machine Learning

shahrinamin_my — 2019-03-08 16:40:38 UTC

Unsupervised Learning discover patterns/relationship from the data itself
- Clustering
  - discover the inherent groupings in the data.
  - E.g. K-means clustering
- Association
  - discover rules that describe large portions of your data.
  - E.g. Apriori algorithm for association rule mining
Supervised Learning based on "training examples". Has the ability to reach an accurate conclusion when given new data.
- Classification
  - predicts an output variable as a category.
  - E.g. Decision Trees, Support Vector Machine.
- Regression
  - predicts an output variable as a real value.
  - E.g. Linear Regression

Reinforcement Learning

shahrinamin_my — 2019-03-08 16:55:04 UTC

The machine trains itself continually using trial and error. It learns from past experience and capture the best possible knowledge to make accurate business decisions.

E.g. Markov Decision Process

Euclidean Distance

shahrinamin_my — 2019-03-08 17:04:57 UTC

calculates the distance between p and q, where each observation has n variables.

K-means Clustering

shahrinamin_my — 2019-03-08 17:08:33 UTC

Randomly pick k number of points for each cluster (centroids)
Each data point is assigned to the cluster with the closest centroids.
Compute the centroid of each cluster based on existing cluster members, this is result in new centroids.
As we have new centroids, repeat step 2 and 3. Repeat this process until convergence occurs. I.e. centroids does not change.

Ez explanation

Randomly pick k number of centroids for each cluster.
Each data is group into nearest centroids.
Move the centroid to the mean of the group.
Repeat 2 and 3 until centroid does'nt change

Limitations

Predefined number of clusters
- self explanatory
Distorted by outliers
- can result in non-optimal grouping. (outlier influence the mean value)
No hierarchical organization

Applications

Marketing
- discover distinct groups in their customer bases.
Insurance
- identify groups of insurance policy holders with a high average claim cost.
City-planning
- identify groups of houses according to their house type, value, and geographical location.
DNA sequences
- cluster DNA information based on edit distance

Association Rule Mining

shahrinamin_my — 2019-03-08 17:21:57 UTC

Support
- How popular an itemset is per all the transactions.
- E.g. Cereal and milk appear together in 40% of the transactions i.e Support = 40%
Confidence
- How likely item Y is purchased when item X is purchased (X->Y)
- E.g. Cereal (X) appear in 50 transactions; 40 of the 50 also include milk (Y). Cereal(X) implies milk(Y) with 80% confidence.
Lift
- Like confidence but while controlling how popular item Y is.
- Lift > 1: Item Y is likely to be bought if item X is bought.
- Lift < 1: Item Y is unlikely to be bought if item X is bought.
- Lift = 1: Those two item are independent of each other. referto
- E.g. Orange milk have 75% confidence, 30% support. Customers generally buy milk 90% of the time and orange 40% of the time.

Lift = 0.3/(0.4*0.9) = 0.83

Phases: Building & Applying

shahrinamin_my — 2019-03-09 14:26:05 UTC

Building
- Data used contains values for the descriptor(X) and response(Y) variables.
- Training set
  - used for building the model.
- Test set
  - accessing the quality of the model
Applying
- Once model has been built, a dataset with no output response variables can be fed into the model to obtain estimated response.

Definition

muzafi17 — 2019-03-09 14:28:38 UTC

Data Wrangling = Transformation of raw data to another form that is useful generating actionable insights
Data Cleaning = To determine inaccurate, incomplete or unreasonable data and then the detected errors are corrected by replacing, modifying or deleting the dirty data
Data Integration = Integrate data from multiple views

Physical - Copy data to warehouse
Virtual - Keep the data only at the sources

Data Reduction = Obtain a reduced data set Dimensionality R - reduce number of random variables or attributes, Numerosity R- replace original data volume by alternative, Data compression - reduce size but preserve its representation
Data Transformation = Normalization :

Min-max - Change to a new min and max
Z-score - Shift of values according to data distribution
Decimal - Move to decimal places to keep max value < 1

Discretization : Divide the range of continuous attribute into intervals because some algorithm accept categorical
Binning - partition into equal size, smoothing by bin means, smoothing by bin boundaries
Entropy-based - find the best split to divide into 2 sets

Info (s1,s2) = |s1|/|s| x E(s1) + |s2| / |s| x E(s2)
Gain(v,s2) = E(s) - info (s1,s2)
H bawah ni E

Tables

shahrinamin_my — 2019-03-09 14:38:29 UTC

You want to show individual data values, and allow audience to compare between individual values.
Precise values are required.
The quantitative information to be communicated involves more than one unit measure.

Graph

shahrinamin_my — 2019-03-09 14:43:34 UTC

You want to reveal relationships between multiple values.
General patterns are the key point you want to present, rather than the exact data values.

Seven Common Relationship in Graphs

shahrinamin_my — 2019-03-09 14:46:34 UTC

Types Of Transformation

muzafi17 — 2019-03-09 14:52:11 UTC

Filtering = Based on some conditions
Transforming = Add new variables
Aggregating = Collapse many to one
Sorting = Change the order of values

Time-Series

shahrinamin_my — 2019-03-09 14:54:46 UTC

Description
- Multiple instances of one or more measures taken at equidistant points in time.
Methods
- Lines to emphasize overall pattern
- Bars to emphasize individual values
- Points connected by lines to slightly emphasize individual values while still highlighting the overall pattern.
- Always place time on the horizontal axis

Nominal Comparison

shahrinamin_my — 2019-03-09 14:57:24 UTC

Description
- A simple comparison of the categorical subdivisions of one or more measure in no particular order.
Methods
- Bars only (horizontal or vertical)

Ranking

shahrinamin_my — 2019-03-09 15:01:46 UTC

Description
- Categorical subdivisions of a measure ordered by size (either descending or ascending)
Methods
- Bars only (horizontal or vertical)
- To highlight high values, sort in descending order.
- To highlight low values, sort in ascending order.

Part-to-Whole

shahrinamin_my — 2019-03-09 15:05:30 UTC

Description
- Measures of individual categorical subdivisions as ratios to the whole.
Methods
- Bars only (horizontal or vertical)
- Use stacked bars only when you must display measures of the whole as well as the parts.

Needs Of Integration = Heterogeneity among data sources

muzafi17 — 2019-03-09 15:10:46 UTC

Source type H Different store systems
Communication H some have web interfaces some offer API
Schema H Different table structures
Data type H Different data type stored
Value H Different ways of same values
Semantic H Different sources different things

Deviation

shahrinamin_my — 2019-03-09 15:12:58 UTC

Description
- Categorical subdivisions of a measure compared to a reference measure, expressed as the differences between them.
Methods
- Lines to emphasize the overall pattern only when displaying deviation and time-series relationships together.
- Points connected by lines to slightly emphasize individual data points while also highlighting the overall pattern when displaying deviation and time-series relationships together.
- Bars to emphasize individual values, but limit to vertical bars when a time-series relationship is included.

Basic Heuristic Methods

muzafi17 — 2019-03-09 15:18:48 UTC

Stepwise forward selection - Start empty, best attribute picked
Stepwise backward selection - Start full, eliminate the worst
Combination - Select best and remove worst
Decision Tree Induction

Frequency Distribution

shahrinamin_my — 2019-03-09 15:51:56 UTC

Description
- Counts of something per categorical subdivisions (intervals) of quantitative range.
Methods
- Vertical bars to emphasize individual values (histogram)
- Lines to emphasize the overall pattern (frequency polygon)

Softwares

muzafi17 — 2019-03-09 15:57:27 UTC

DataWatch Monarch
Trifacta Wrangler
Openrefine

Correlation

shahrinamin_my — 2019-03-09 15:58:38 UTC

Description
- Comparisons of two paired sets of measures to determine if one set goes up, the other set goes either up or down in a corresponding manner, and it so, how strongly.
Methods
- Points and trend line in the form of a scatter plot.
- Bars may be used, arranged as a paired bar graph or correlation bar graph, if scatter plots are unfamiliar.

taktau ah noto

muzafi17 — 2019-03-09 16:00:27 UTC

Baca dan faham analysis

muzafi17 — 2019-03-09 16:10:37 UTC

No final

muzafi17 — 2019-03-09 16:11:13 UTC

Tools/Languages for DS

shahrinamin_my — 2019-03-09 16:15:46 UTC

R (statistics)
Python (pandas, numpy)
Scala (enable data streaming)
SQL (RDBMS)
Excel (data analysis)
Java
Matlab (academic/math)
SPSS (statistical packages)
WEKA (machine learning)
RapidMiner (DS software platform)
Hadoop (process extremely large data sets)
D3.js (data visualization)
Tableau (data visualization)
Apache Spark (machine learning)
SAS (analytics software)

Structured Data

muzafi17 — 2019-03-09 16:26:10 UTC

High degree of organisation such as a relational database

Unstructured

muzafi17 — 2019-03-09 16:27:41 UTC

Information that is difficult to organise using traditional mechanics

Python vs R for Data Science

shahrinamin_my — 2019-03-09 16:33:00 UTC

link

Python
- Advantages
  - Improvement in availability of Python packages.
  - A good tool to implement algorithms for production use.
R
- Advantages
  - Easier for beginner to do exploratory work.
  - Usable for basic data analysis without installation of packages

SD vs UD

muzafi17 — 2019-03-09 16:40:32 UTC

Data Growth

muzafi17 — 2019-03-09 16:53:00 UTC

UD 60-80% per year
1 exabytes = 1000 petabytes = 1 million terabytes = 1 billion gigabytes

SQL Database

muzafi17 — 2019-03-09 16:56:34 UTC

Relational database (RDBMS)
Table based database
Web, mobile, enterprise, data mart

NoSQL Database

muzafi17 — 2019-03-09 16:57:14 UTC

Non-relational/distributed database
Document based, key-value pairs, graph database or wide-column stores
Gaming, social, iot, web

ACID Compliancy

muzafi17 — 2019-03-09 17:10:15 UTC

Atomicity = All data and commands in a transaction succeed, or all fail and roll back
Consistency = All committed data must be consistent with all data rules including constraints, triggers, cascade, atomicity, isolation and durability
Isolation = Other operations cannot access data that has been modified during a transaction that has not yet completed
Durability = Once a transaction is committed, data will survive system failures, and can be reliably recovered after an unwanted deletion

BASE

muzafi17 — 2019-03-09 17:19:26 UTC

Basically Available = Guaranteed availability
Soft-state = The state of the system may change, without even a query because of node updates
Eventually consistent = The system will become consistent over time

Relational Database

muzafi17 — 2019-03-09 17:49:46 UTC

Use the notion of databases separated into tables where each column represents a field and each row represents a record

MySQL
PostgreSQL
SQLite3

Normalisation of Relational DB

muzafi17 — 2019-03-09 17:56:03 UTC

1NF : Eliminate groups of repeating data by creating a newtable for each group of related data which is identified by a primary key
2NF : If a set of values are the same for multiple records move them to a new table and link the two tables with a foreign key
3NF : Fields which do not depend on the primary key of a table must be removed and if necessary be put into another table

ACID Properties of Relational DB

muzafi17 — 2019-03-09 18:05:28 UTC

Atomicity : Either all parts of a transaction must be completed or none
Consistency : The integrity of the database is preserved by all transactions. The database is not left in an invalid state after a transaction
Isolation : A transaction must be run isolated in order to guarantee that any inconsistency in the data involved does not affect other transactions
Durability : The changes made by a completed transaction must be preserved or in other words be durable

Non-Relational Database

muzafi17 — 2019-03-09 18:15:31 UTC

Represent data in collection of JSON docs
Records can have different fields (Dynamic Schema)

Key-value stores Redis Voldemort Dynamo
Column-oriented database Cassandra Hbase link
Document-based stores MongoDB link
Graph databases Neo4J InfiniteGraph link

Advantages Of NoSQL

muzafi17 — 2019-03-09 18:34:29 UTC

Generally process data faster than relational DB
Faster because their data models are simpler
No schema required

Redis

muzafi17 — 2019-03-09 18:39:11 UTC

Best used: For rapidly changing data with a foreseeable database size
Use cases: To store real-time stock prices, Real-time analytics. Leaderboards. Real-time communication

Cassandra

muzafi17 — 2019-03-09 18:55:56 UTC

Advantage of Column-oriented = Some types of data lookups can become very fast, given that the desired data could be stored consecutively in a single row
Best used: When you need to store data so huge that it doesnt fit on server, but still want a friendly familiar interface to it
Use cases: Web analytics, to count hits by hour, by browser, by IP. Transaction logging. Data collection from huge sensor arrays

MongoDB

muzafi17 — 2019-03-09 19:04:28 UTC

Load balancing, replication, indexing, querying and act as file system
Best used: If you need dynamic queries. If you need good performance on big DB
Use cases: For most things that you would do with MySQL or PostgreSQL but having predefined columns

Neo4J

muzafi17 — 2019-03-09 19:07:21 UTC

Advantage of Graph-based = Potential in certain data mining and pattern recognition scenarios
Best used: For graph-style, rich or complex, interconnected data
Use cases: For searching routes in social relations, public data transport links, road maps, or network topologies

Python for Data Analysis

shahrinamin_my — 2019-03-10 02:46:15 UTC

After this thing, just for reading/understanding only. Never ask one. Just know the basic coding using pandas & numpy.

Numpy (Numerical Python)

shahrinamin_my — 2019-03-10 02:47:39 UTC

A fast and efficient multidimensional array object ndarray
Tools for integrating C, C++ and Fortran code for Python
More efficient way of storing and manipulating data (numerical data).

Pandas

shahrinamin_my — 2019-03-10 02:56:16 UTC

The primary object in pandas is the DataFrame, a two-dimensional tabular, column oriented data structure with both row and column tables.

*this sht have a lot of stuff to read

Pandas Cheat Sheet!

Matplotlib

shahrinamin_my — 2019-03-10 03:08:00 UTC

Python library for producing plots and other 2D data visualizations.

Bridging the R-Python Divide

shahrinamin_my — 2019-03-10 03:10:52 UTC

Python package rpy2 allows one to call R from within Python

Can translate Python objects into R, pass into R functions, and convert the R output back into Python objects.

Python -> R functions -> R output -> Python

Trimester 2, 2016/17

shahrinamin_my — 2019-03-10 03:28:24 UTC

Question 1
- Key-enablers (1)
- 5V's (1)
- Challenges in DS (1)
- Data structures (1)
Question 2
- Types of questions (2)
- Heterogeneity (4)
- Data Quality Dimensions (2)
- Errors/Inconsistencies (4)
Question 3
- Python code (5)
- Correlation (6)
- Machine learning (7)
Question 4
- SQL and NoSQL (11)
- Python is the preferred language (5)
- Appropriate Chart/Graph (9)

Chapter 1, 2, 4, 5, 6, 7, 9 ,11
Yg x keluar, 3,8,10,12

Trimester 1, 2017/18

shahrinamin_my — 2019-03-10 04:03:21 UTC

Question 1
- Responsiblities of Data Scientist (1)
- 5V's (1)
- Big Data Insights (1)
- Characteristics of Big Data (1)
Question 2
- Types of questions (2)
- Outliers (6)
- Data Quality Dimensions (2)
- Errors/Inconsistencies (4)
Question 3
- Python code (5)
- Machine learning (7)
- Correlation (6)
Question 4
- SQL and NoSQL (11)
- R is the preferred language (5)
- Graph/Table (9)

Chapter 1, 2, 4, 5, 6, 7, 9, 11
Yg x keluar, 3,8,10,12

Trimester 2, 2017/18

shahrinamin_my — 2019-03-10 04:12:31 UTC

Question 1
- BI vs DS (1)
- Responsibilities of Data Scientist (1)
Question 2
- Entropy-based (4)
- Calculate Gain (4)
- NoSQL (11)
- MongoDB (11)
- Python packages contains R (5)
Question 3
- Training set and Test Set (8)
- Association Rules Mining - Confidence, Lift (7)
- Euclidean Distance (7)
Question 4
- Python Code (5)
- Graph/Table (9)

Chapter 1, 4, 5, 7, 8, 9, 11
Yg x keluar, 2, 3, 6, 10 ,12

Pick Rate

shahrinamin_my — 2019-03-10 04:22:39 UTC

100%

Chapter 1, 4, 5, 7, 9, 11

67% (Twice)

Chapter 2, 6

33% (Once)

Chapter 8

0% (Never)

Chapter 3, 10, 12

Most Favorite Questions

5V's (1)
Responsibilities of Data Scientist (1)
Type of Questions (1)
Malas da. lol
Buat sendiri
Bapak ar
Pahal kau
Jahat nya kamu
k

Discrepancies in Data

muzafi17 — 2019-03-10 04:34:37 UTC

1. Incomplete data

Non available value
Different criteria between time data collected when it is analysed
Human/hardware/software problems

Ignore the tuple
Fill in missing value manually
Use a global constant to fill
Use the attribute mean/median Predict using a learning algorithm

2. Noisy data

Faulty instruments
Data entry or computer errors
Data transmission

Binning methods
Outlier analysis
Regression
Combination of human and computer inspection

3. Inconsistent / redundant

Different data sources
Functional dependencies
Referential integrity violation

Naming conventions
Parsing text into fields
Different representations
Primary key violation
Formatting issues

Traditional Database vs Modern Database

shahrinamin_my — 2019-03-10 15:45:12 UTC

- Data Sources
Both: MS SQL
T: Tables, Files
M: Social, Streaming

- Data Structure
Both: Structured data
T: ER-model, Multi-dimensional Model
M: Unstructured data, Machine data

- Data Access
T: SQL
M: Parallel processing, Distributed processing, In-memory processing

- Analytics
T: Reporting, Dashboard, Data Analysis
M: Regression, Clustering, Data Mining

Skills in Academia vs Industry?

shahrinamin_my — 2019-03-10 15:45:36 UTC

Academia
- Communication and People Skill
- Industry Skill
- Business Process Skill
+ Data Skill
+ Analytical Skill
+ Technical Skill

Industry
+ Communication and People Skill
+ Industry Skill
+ Business Process Skill
- Data Skill
- Analytical Skill
- Technical Skill

Simpson's Paradox

shahrinamin_my — 2019-03-10 15:48:05 UTC

Simpson Paradox states that the representation of the data will be misleading without considering the confounding factors.

Metrics to evaluate model accur

shahrinamin_my — 2019-03-10 15:49:04 UTC

- Classification accuracy
Sensitivity (Recall) = TP / P
Precision = TP / P'
F1-score
P = all actual positive values
P' = all predicted positive values
F1 = 2 x R x P / (R+P)

Regression metrics:
- MAE
- MSE
-

- Area under the ROC graph

Choropleth

shahrinamin_my — 2019-03-10 15:50:49 UTC

to represent values on a map with different shade of colour to represent the value.

Refer Refer

shahrinamin_my — 2019-03-11 09:10:24 UTC

https://en.wikipedia.org/wiki/K-nearest_neighbors_algorithm