Is the baseline current practice just the simplest model where we get a value that we can compare our model with? Or does it mean that we use Company X's current practice? In HW1 the current practice was given, but this time it sounds like they just don’t have a current practice, so we're a bit confused.

Answer: An example mentioned in class is that someone walks into the doctor's office and that doctor tests everyone for breast cancer. I emphasized how that is in itself making a prediction. The same thing applies here. 

To answer your two questions with a question mark: It means that you use the Company X's current practice. 

In our (last) review class, it's mentioned that often the current baseline has to be inferred and figured out and is not explicitly stated as in HW1. In real life, that's almost always the case.

Answer 1: Yes, that’s right.

Question 2: Is it the baseline model in a formula sort of way like you mentioned in lecture 11.  

Answer 2: The point of the formula was different. it was to say that sometimes a profit matrix is to be specified as a constant value, or as depending on other parameters. it's for you to figure out which applies for the baseline in this case.
Question 3: If it is the confusion matrix method , Should we :
1.logically assume values that go into the confusion matrix ? or

 2.should we go with some sort of numbers from within the assignment ? 

Answer 3: Use values from the assignment, and where you need to make an assumption, make it and state it.

The question asks how you would preprocess these data, to extract more meaningful information/signal from the dataset. So the key point is how you would deal with the data issue and make it make sense.

Adjusting the dataset is one way to do it, you can also deal with it in other ways as long as it makes sense.

It would be better if you can present how you preprocess the data instead of how the data issues can affect the model.

Also, In thes slides, basically you have upsampled the minority class and downsampled the majority class to make it equal (500/500). Should we also do both? or is doing one enough? 

Also, what is the safe ratio that we are looking for when we do any of these?

Answer: As you mentioned, we can upsample the minority class or downsample the majority class to make it equal (500/500), so you can choose one of them and don’t need to do both.

I didn’t fully understand the question “why did you only downsample only 10% of the data in the DR example?” and “safe ratio”. I presume you mean the downsampling we did in DataRobot, where we resampled an extremely imbalanced European Credit Card dataset to 1:9 (10%). Here the 10% means the percentage of the minority, and we should resample the training data to make it 5:5. 

With a train-valuation-holdout split, you should resample your training data and get resampled data, so the percentage of the resampling data is the percentage of your training data.

Also, if we are resampling the data, as per Lecture 10 (Page 20), we should evaluate the model on the initial train data set. I have a doubt that when we should use the test data set. Can you please clarify that as well?

Answer 1: Yes, you are supposed to get the total expected value through the profit matrix. 

The value of ‘new charges’ is not provided in the raw data, so your task is to make simplifying assumptions to assume values for these missing variables based on some domain knowledge of bank customer spending. Given all the previous bill statements for the person that we see, what would likely be missing values in the FN cell? Making reasonable assumptions is part of data science (machine learning for business), if you could do so, explain it, and measure the elements in the profit matrix, then I think you can find the way to calculate the ‘total expected value of the action and model’ (see lecture 12 videos). You're graded on coming up with a reasonable approximation. There's not just one solution, so choose the best approximation you can think of.

In HW3, you should be able to combine resampling and use the train-valuation-holdout (Valuing Action on a 3rd fold) approach we taught. The best model trained on the resampled data would be your Model* (see lecture 12 videos). Then, you VALUE your model* on the valuation fold to find the best action* (e.g. threshold; watch lecture 12) with given model*. Next, you take the best model* and action* and report the value on the holdout set. The test data set is the holdout set, and it will be used in the end to evaluate the performance of Model* with action*.

Question 2: Q3.a.iii.1. says "Are there qualitative elements you cannot measure? If so, mention them, and explain how this will affect your eventual model valuation."

Since it says "eventual model valuation", aren't we supposed to get the confusion matrix and profit matrix for the holdout set (after valuation) in earlier questions 3.a.i & ii?

Answer 2: In Q3, you can show a confusion matrix for any of the full dataset, valuation set or holdout set.

Also, Q3 asks you to specify the "qualitative elements in each cell of the profit matrix" (which can be static), and then specify how you would "calculate numeric values in each profit matrix cell" (the formula would be static). Throughout lecture 11, it’s mentioned that some profit matrices would consist of numbers, some would consist of an equation to calculate profit cell matrix values. In Q3 you are able to show the equation of each cell, and then describe how you calculate it. 

So you can have the confusion matrix and profit matrix for any of the full dataset, valuation set or holdout set, and you can show one of the three datasets.