# data[].attributes.headline
Positive Cases (Left-side chart) vs. Normalized Positive / totalTestResult (Right-side chart) By Location

# data[].attributes.subject
Positive Cases (Left-side chart) vs. Normalized Positive / totalTestResult (Right-side chart) By Location

# data[].attributes.body
Before Normalized : California looks more severe (280K +ve) than Connecticut ( 36K +ve)
After Normalized : California (-236) and Connecticut (-200) is rather similar in terms of severity

# data[].attributes.body
The normalization technique transform the skewed distribution to a normal distribution

# data[].attributes.body
Normalizing Positive Cases, Using totalTestResult as baseline denominator

# data[].attributes.headline
Histogram For totalTestResult & Positive Cases

# data[].attributes.subject
Histogram For totalTestResult & Positive Cases

# data[].attributes.body
Skewed Distribution for totalTestResults('blue') & positive('red')

# data[].attributes.attachment_link.title
Set Based Value analyses using Bar Chart - Image 2

# data[].attributes.headline
Data Profiling on single column - Missing values

# data[].attributes.subject
Data Profiling on single column - Missing values

# data[].attributes.body
<<to describe data publish tasks performed>>

The Data Canvas in TIBCO was "run" and saved as CSV in our library.

# data[].attributes.body
<<to describe the data cleaning or transformation tasks performed>>

Purpose:

1. To identify and remove data quality problems.

2. To transform the data to produce a good dataset for analytics activities.

Techniques Used:

For Invalid Data (By Kuang)

The current date is set as ‘1/1/2020’ which is inaccurate as after ‘31/5/2020’ and before ‘2/6/2020’ is supposed to be 1/6/2020’. Therefore, the value is replaced as ‘1/6/2020’ and is applied to only this occurrence.

It is necessary to replace the inaccurate values with the appropriate dates in chronological order as shown so that the dates can be visualised accurately

For Redundant Column (By Kuang)
The column ‘Total’ is deleted by performing destructive transformation to remove the redundant column.

The data dictionary defines the ‘Total’ column as the total number of positive, negative and pending test results. However, it shows that there are ‘0’ tests conducted indicated in the column. Therefore, it is necessary to transform the dataset to remove redundant columns which will not help in the visualisation of data.

Data Unpivot using TIBCO (By Junaidah)

3 columns in a wide data shape were selected. We have transformed them into tall and skinny data using unpivot function in TIBCO Spotfire. This is more suitable for analysis and modeling in our subsequent steps

Transform data based on states, date, tested positive and categorized into recovered, death, hospitalised currently and on ventilator currently. This is to highlight on the current information and up on per day basis.

Data Normalization (By Tze Wee)

Based on WHO's recommendation, testing is one of the most important tools in controlling an outbreak. When universal testing is implemented, people who are infected with the virus can be isolated from folks who test negative. This functions as a targeted social distancing technique and can help slow the outbreak.

Thus, we decided to compare the test results vs. cases tested positive.

However,

1. these 2 columns are of very different unit of measures.

2. Distribution is not normal and skewed to 1 side (see Histogram 'blue' & 'red').

Log10 Normalization Technique was applied

Log10 Normalization of Positive Cases

Using totalTestResults as a baseline denominator

Results:

1. Combine distribution are of a normal distribution for further application (see Histogram 'orange').

2. Analysis of Positive Cases in each location, normalized by the total test results are more insightful, for more accurate comparison.

# data[].attributes.body
<<to describe the data profiling tasks performed>>

For the dataset selected, we recognised that we need to use augmented approach. We will be using TIBCO to analyse and visualise the dataset.

We will use both Single column and Multi columns profiling.

For Single column profiling, we will use look at Cardinality data ie on missing values.

For Multi columns profiling, we will look at Dependency Check on Partial dependency and Conditional dependency data ie on positive, hospitalized, on ventilator, recovered, death and total number of test results.

We will use Set-based value analyses using numerical value ie using Bar Chart

Purpose:

To understand the available data, and data quality problem, if any.

Techniques Used:

Automated data profiling identified columns with missing data, of which 2 columns are selected, by each member of our group

# data[].attributes.body
<<to describe the join or append data tasks performed (if any)>>

No Blending was done. Perhaps when new data is available, we can apply union join to add in the new data, to the main dataset.

# data[].attributes.body
<<to describe the data sources>>

To access the data from the data sources from a public source https://www.kaggle.com/sudalairajkumar/covid19-in-usa.

The dataset contains data for all Covid‐19 cases in USA, from Jan to Sep 2020. They include information such as date, state/region, number of hospitalized, number of new cases, etc.

Technique Used:

Load the data using TIBCO Spotfire.

# data[].attributes.body
<<to describe the project and data preparation problems>>

The objective of our project is to apply what we are taught, applying various data augmentation tools, to understand the Covid-19 situation in USA, the largest economy in the world. How well this health crisis is manage and pent out in the USA is of utmost importance, as it will have a direct impact on how the rest of the world will recover, including Singapore.

We hope to be able to gain insights from our data analysis, after our data preparation and data profiling, for us to have a more objective perspective of this health crisis.

And, we hope at the end of this project, we will gather sufficient, vital information to comprehend the complexity and magnitude of this dire health situation, be able to identify the trend lines of where this pandemic are heading and hopefully identify possible solutions to mitigate the severity of this catastrophe, giving us some hope, how we will recover our lives, from the health crisis of our generation, back to normalcy.

In order to achieve our objective, we need to prepare our data by;

1. Gaining Access To The data (datasets)

2. Identify the Data Quality Problems

Invalid Values (Inconsistencies, Outliers, Noise, Multiple Variables In 1 Column)

Duplication

3. Perform Data Cleaning/Transformation to Improve The Data Quality