We hope to be able to gain insights from our data analysis, after our data preparation and data profiling, for us to have a more objective perspective of this health crisis. 

And, we hope at the end of this project, we will gather sufficient, vital information to comprehend the complexity and magnitude of this dire health situation, be able to identify the trend lines of where this pandemic are heading and hopefully identify possible solutions to mitigate the severity of this catastrophe, giving us some hope, how we will recover our lives, from the health crisis of our generation, back to normalcy. 

In order to achieve our objective, we need to prepare our data by;

The dataset contains data for all Covid‐19 cases in USA, from Jan to Sep 2020. They include information such as date, state/region, number of hospitalized, number of new cases, etc.

Technique Used:

Load the data using TIBCO Spotfire.

For Single column profiling, we will use look at Cardinality data ie on missing values. 

For Multi columns profiling, we will look at Dependency Check on Partial dependency and Conditional dependency data ie on positive, hospitalized, on ventilator, recovered, death and total number of test results.

We will use Set-based value analyses using numerical value ie using Bar Chart 

Purpose:

To understand the available data, and data quality problem, if any.

Techniques Used:

Automated data profiling identified columns with missing data, of which 2 columns are selected, by each member of our group

Techniques Used:

For Invalid Data (By Kuang)

The current date is set as ‘1/1/2020’ which is inaccurate as after ‘31/5/2020’ and before ‘2/6/2020’ is supposed to be 1/6/2020’. Therefore, the value is replaced as ‘1/6/2020’ and is applied to only this occurrence.

It is necessary to replace the inaccurate values with the appropriate dates in chronological order as shown so that the dates can be visualised accurately

For Redundant Column (By Kuang)
The column ‘Total’ is deleted by performing destructive transformation to remove the redundant column.

The data dictionary defines the ‘Total’ column as the total number of positive, negative and pending test results. However, it shows that there are ‘0’ tests conducted indicated in the column. Therefore, it is necessary to transform the dataset to remove redundant columns which will not help in the visualisation of data.


Data Unpivot using TIBCO (By Junaidah)

3 columns in a wide data shape were selected. We have transformed them into tall and skinny data using unpivot function in TIBCO Spotfire. This is more suitable for analysis and modeling in our subsequent steps

 Transform data based on states, date, tested positive and categorized into recovered, death, hospitalised currently and on ventilator currently. This is to highlight on the current information and up on per day basis.


Data Normalization (By Tze Wee)

Based on WHO's recommendation, testing is one of the most important tools in controlling an outbreak. When universal testing is implemented, people who are infected with the virus can be isolated from folks who test negative. This functions as a targeted social distancing technique and can help slow the outbreak.

Thus, we decided to compare the test results vs. cases tested positive. 

However,