# data[].attributes.headline
I want to check if I'm thinking correctly:  If the customer spends 6000 on new transaction, did they actually buy something for 5 820 and they pay 180 on transactions?

# data[].attributes.subject
I want to check if I'm thinking correctly:  If the customer spends 6000 on new transaction, did they actually buy something for 5 820 and they pay 180 on transactions?

# data[].attributes.body
If the customer spends 6000 on new transaction, then they actually buy something for 6000. The transaction fee 180 will be paid by the merchant.

# data[].attributes.body
Should we use confusion matrix for "suspend spending privileges" in 3a or baseline model? Since it asks "For the action in (1.a), specify the following valuation models" in the question we think we should use c.matrix for the action (suspend spending privileges). Is that right?

# data[].attributes.body
Question 1: I have a question regarding HW3. In the 8th question, it asks the ‘total expected value of the action and model’. Are we supposed to get the total profit through the profit matrix or just an explanation would be enough? (Because we can’t get the value in FN cell as we don’t know the value of new transactions to calculate the ‘fee on new charges’)

Also, if we are resampling the data, as per Lecture 10 (Page 20), we should evaluate the model on the initial train data set. I have a doubt that when we should use the test data set. Can you please clarify that as well?

Answer 1: Yes, you are supposed to get the total expected value through the profit matrix.

The value of ‘new charges’ is not provided in the raw data, so your task is to make simplifying assumptions to assume values for these missing variables based on some domain knowledge of bank customer spending. Given all the previous bill statements for the person that we see, what would likely be missing values in the FN cell? Making reasonable assumptions is part of data science (machine learning for business), if you could do so, explain it, and measure the elements in the profit matrix, then I think you can find the way to calculate the ‘total expected value of the action and model’ (see lecture 12 videos). You're graded on coming up with a reasonable approximation. There's not just one solution, so choose the best approximation you can think of.

In HW3, you should be able to combine resampling and use the train-valuation-holdout (Valuing Action on a 3rd fold) approach we taught. The best model trained on the resampled data would be your Model* (see lecture 12 videos). Then, you VALUE your model* on the valuation fold to find the best action* (e.g. threshold; watch lecture 12) with given model*. Next, you take the best model* and action* and report the value on the holdout set. The test data set is the holdout set, and it will be used in the end to evaluate the performance of Model* with action*.

Question 2: Q3.a.iii.1. says "Are there qualitative elements you cannot measure? If so, mention them, and explain how this will affect your eventual model valuation."

Since it says "eventual model valuation", aren't we supposed to get the confusion matrix and profit matrix for the holdout set (after valuation) in earlier questions 3.a.i & ii?

Answer 2: In Q3, you can show a confusion matrix for any of the full dataset, valuation set or holdout set.

Also, Q3 asks you to specify the "qualitative elements in each cell of the profit matrix" (which can be static), and then specify how you would "calculate numeric values in each profit matrix cell" (the formula would be static). Throughout lecture 11, it’s mentioned that some profit matrices would consist of numbers, some would consist of an equation to calculate profit cell matrix values. In Q3 you are able to show the equation of each cell, and then describe how you calculate it.

So you can have the confusion matrix and profit matrix for any of the full dataset, valuation set or holdout set, and you can show one of the three datasets.

# data[].attributes.headline
For question 4 is it hypothetical or should we actually create a model? So then we would have two models one for classification and one for regression predictions and our baseline?

# data[].attributes.subject
For question 4 is it hypothetical or should we actually create a model? So then we would have two models one for classification and one for regression predictions and our baseline?

# data[].attributes.body
No, you don’t have to create a regression model in Q4. The question is “How would you prepare the data for use in a regression model?”, so you only need to clarify how you prepare the data for the regression model with the raw data.

# data[].attributes.body
Question: Hello, we are currently looking at the dataset for the ML exam. We have noticed several enological values in the dataset. for example there are several customers with 0 bill AMT and Pay AMT but still labeled as default next month, we also have customers with the same values which are not labeled as defaults next month. Further the Pay column seem inconsistent where there are several -2 values that should be -1 and opposite. We are thinking of how to deal with this, either we try and adjust the dataset to remove most of these inconsistencies or should we take the dataset as it is and just include a statement in our task discussing the inconsistencies in the data and how that can affect the model.
Answer: You are right, the dataset is imperfect and there are some data inconsistencies. I guess you are asking question related to Q6. So you should specify the data issues you notice, state any recoding/preprocessing you think would be useful to deal with the data, and think whether deriving new features could be useful.

The question asks how you would preprocess these data, to extract more meaningful information/signal from the dataset. So the key point is how you would deal with the data issue and make it make sense.

Adjusting the dataset is one way to do it, you can also deal with it in other ways as long as it makes sense.

It would be better if you can present how you preprocess the data instead of how the data issues can affect the model.

# data[].attributes.body
Question: When there is imbalance and we decide to downsample for instance. Do we make the majority class equal to the minority class as you had shown in class/slides? Also, then why did you only downsample only 10% of the data in the DR example, is the dataset still not imbalanced?

Also, In thes slides, basically you have upsampled the minority class and downsampled the majority class to make it equal (500/500). Should we also do both? or is doing one enough?

Also, what is the safe ratio that we are looking for when we do any of these?

Answer: As you mentioned, we can upsample the minority class or downsample the majority class to make it equal (500/500), so you can choose one of them and don’t need to do both.

I didn’t fully understand the question “why did you only downsample only 10% of the data in the DR example?” and “safe ratio”. I presume you mean the downsampling we did in DataRobot, where we resampled an extremely imbalanced European Credit Card dataset to 1:9 (10%). Here the 10% means the percentage of the minority, and we should resample the training data to make it 5:5.

With a train-valuation-holdout split, you should resample your training data and get resampled data, so the percentage of the resampling data is the percentage of your training data.

# data[].attributes.body
Hi,
Is next month's default amount meaning in Q4 is the predicted BILL Amount on October*prediction value from classification model?

# data[].attributes.headline
A follow-up question from students about Q4

# data[].attributes.subject
A follow-up question from students about Q4

# data[].attributes.body
Question: How to interpret the negative values? Are the negative values in the dataset people who have overpaid their account or is negative when someone else pays that persons balance. And if it is someone paying for them how do we know their actual amount owed?
Answer: The negative value means cardholders overpaid their account. 5000 and -2000 are the balances of two different cardholders. The hint asks whether predicting negative values (e.g. -2000) would make any sense. So you can think about this hint when answering Q4.

# data[].attributes.headline
Questions about confusion matrix basline model

# data[].attributes.subject
Questions about confusion matrix basline model

# data[].attributes.body
Question 1: We are to create a confusion matrix basline model , isnt it ?

Answer 1: Yes, that’s right.

Question 2: Is it the baseline model in a formula sort of way like you mentioned in lecture 11.

Answer 2: The point of the formula was different. it was to say that sometimes a profit matrix is to be specified as a constant value, or as depending on other parameters. it's for you to figure out which applies for the baseline in this case.
Question 3: If it is the confusion matrix method , Should we :
1.logically assume values that go into the confusion matrix ? or

2.should we go with some sort of numbers from within the assignment ?

Answer 3: Use values from the assignment, and where you need to make an assumption, make it and state it.

# data[].attributes.body
Hi,
In the last question we have to do a valuation of the model. I've watched lecture 12 but that dataset is very different from what we have. Is there any other resources that would be helpful when doing the valuation?

# data[].attributes.headline
Question about "current baseline/practice" 

# data[].attributes.subject
Question about "current baseline/practice" 

# data[].attributes.body
Question: It says in the assignment that Company X is currently not predicting default, and we're then confused on the meaning behind current baseline if they're not actually predicting anything as "current practice".

Is the baseline current practice just the simplest model where we get a value that we can compare our model with? Or does it mean that we use Company X's current practice? In HW1 the current practice was given, but this time it sounds like they just don’t have a current practice, so we're a bit confused.

Answer: An example mentioned in class is that someone walks into the doctor's office and that doctor tests everyone for breast cancer. I emphasized how that is in itself making a prediction. The same thing applies here.

To answer your two questions with a question mark: It means that you use the Company X's current practice.

In our (last) review class, it's mentioned that often the current baseline has to be inferred and figured out and is not explicitly stated as in HW1. In real life, that's almost always the case.

# data[].attributes.headline
Question 1 states that suspending spending privileges can be an action/intervention, then in question 3 after putting up the profit matrix asks " Are there qualitative elements you cannot measure? If so, mention them, and explain how this will affect your eventual model valuation". We then thought to mention the administrative efforts/cost, because measures like customer satisfaction would be difficult to quantify without experts. Should we look away from this?

# data[].attributes.subject
Question 1 states that suspending spending privileges can be an action/intervention, then in question 3 after putting up the profit matrix asks " Are there qualitative elements you cannot measure? If so, mention them, and explain how this will affect your eventual model valuation". We then thought to mention the administrative efforts/cost, because measures like customer satisfaction would be difficult to quantify without experts. Should we look away from this?

# data[].attributes.body
Answer: the question states "there is no cost/benefit associated with customer goodwill" before it specifies question 1. That means assume that. So you can disregard customer goodwill/dissatisfaction costs completely. Question 3.ii then asks you to specify qualitative elements (here, you shouldn't mention customer goodwill, because the earlier assumption asked you to disregard it; Sidepoint: HOWEVER, administrative costs as a qualitative element is a fair answer, as can be independent of customer goodwill; e.g. you may say that if the customer wasn't to default but we suspended their spending privileges, we'd have to send a letter and apologize to them.), and then 3.iii(1) asks for the numeric values in each profit matrix cell, and asks if there is one you cannot measure, then mention it and explain how it will affect your model evaluation. Here you can mention the administrative costs that I mentioned above, and specify how it will affect your model. Just don't forget to specify the numeric values in each profit matrix cell too.

# data[].attributes.headline
If " there is no cost/benefit associated with customer goodwill" or customer dissatisfaction, how can we in question 1 formulate action/interventions.

# data[].attributes.subject
If " there is no cost/benefit associated with customer goodwill" or customer dissatisfaction, how can we in question 1 formulate action/interventions.

# data[].attributes.body
Answer: Actions don't rely on you having customer goodwill; An action that the assignment suggests is "suspending spending privileges", that is the action. If customer goodwill existed (which we've asked you to assume it doesn't), it would have contributed to qualitative elements in the matrix, it wouldn't have been an action itself.
The assignment only expects you to value "suspending spending privileges", but to mention 3 other potential actions and justify your choice (specifying their limitations would be a plus here).

# data[].attributes.headline
We are a bit confused by the dataset. Is the bill columns accumulated or monthly? 

# data[].attributes.subject
We are a bit confused by the dataset. Is the bill columns accumulated or monthly? 

# data[].attributes.body
Answer: The bill columns are accumulated. For example, if BILL_AMT4 JUNE  is 3272 and BILL_AMT3 JULY is 2682, then 2682 NT dollars (but not 3272 + 2682 NT dollars) is the total amount owed to the bank before August. So, 2682 is the accumulated bill by the end of July.

# data[].attributes.headline
Hello! We have several questions for this amazing project :D

# data[].attributes.subject
Hello! We have several questions for this amazing project :D

# data[].attributes.body
1) Is the balance in the ‘bill-amt’ column, the ending balance or the beginning balance?
2) For Q3(iii), do we need to get the final value for the answer or can we just put the generic equation, such as 2% of carried balance instead of the final value.
3) For Q4, to prepare the data for continuous prediction, do we need to add new attributes to the dataset? For example, how much is the actual default for each customer.
4) For Q4, is it possible to consider partial default as ‘default’? Or it is always 100% default. For example, the outstanding balance of customer A is 10,000 NOK and next month he paid only 6,000 NOK. Do we call 4,000 NOK default?