people + places + love + AIRBNB by Noelle https://padlet.com/qiwenlim/f6ov9qemv2dxzb1f Pre-processing web scraped data from Inside Airbnb (http://insideairbnb.com) for machine learning en-us 2020-12-03 11:21:23 UTC 2024-05-19 00:22:18 UTC hello@padlet.com https://padlet-uploads.storage.googleapis.com/894165161/047e332ee0d3862eaa8c1317470454e8/tp_output_onlinejpgtools.png Introduction limhy2009 https://padlet.com/qiwenlim/f6ov9qemv2dxzb1f/wish/989943243 Short-term rentals platforms such as Airbnb have made it easy for homeowners to list their apartments or spare rooms online.

Since Airbnb is a market, the amount a host can charge is ultimately tied to market prices and one challenge that Airbnb hosts face is determining the best price to rent out a space. ]]> 2020-12-05 09:29:24 UTC https://padlet.com/qiwenlim/f6ov9qemv2dxzb1f/wish/989943243 scrutinise the DATA and it will confess to anything limhy2009 https://padlet.com/qiwenlim/f6ov9qemv2dxzb1f/wish/989978386 Data on Airbnb listings in Singapore was obtained from insideairbnb.com, a free to use repository of Airbnb listings in various cities around the world.

Click worksheet names for data dictionaries
Airbnb listings (26 columns, 7908 aggregated rows of tall & skinny property listing data) -- the apartments, their furnishing, prices, and past review ratings, etc.
  • 3700 listings (rows) from hosts who registered from 2009 to 2015 
  • 4208 listings from hosts who registered from 2016 to 2019 

Airbnb neighbourhood (2 columns, 55 rows of categorical labels) -- the towns and regions where the apartments were located.
]]> 2020-12-05 10:19:19 UTC https://padlet.com/qiwenlim/f6ov9qemv2dxzb1f/wish/989978386 machines can See, Hear and Learn limhy2009 https://padlet.com/qiwenlim/f6ov9qemv2dxzb1f/wish/989985068 Machine Learning can be used anywhere from automating mundane tasks to offering intelligent insights.

Predictive analytics are particularly useful and can help Airbnb hosts price their apartments, given the apartments' traits vis-a-vis competing offerings in the market. Examples include:
  • training regression models to predict the target rent rates the hosts should charge, using a basket of predictors like the apartments' locations and nearness to amenities and attractions, the apartment sizes and furnishings
  • making use of classification models (Decision Tree, ANN, etc.) to explore whether the apartment attributes improve (Y/N) rent rates
  • combining the results of various models with Ensemble algorithms for "Wisdom of the Crowd" effects 
]]> 2020-12-05 10:29:02 UTC https://padlet.com/qiwenlim/f6ov9qemv2dxzb1f/wish/989985068 References limhy2009 https://padlet.com/qiwenlim/f6ov9qemv2dxzb1f/wish/990718157 1. Predicting Airbnb prices with machine learning and location data https://towardsdatascience.com/predicting-airbnb-prices-with-machine-learning-and-location-data-5c1e033d0a5a

2. Hands-on with Feature Engineering Techniques: Transforming Variables
https://heartbeat.fritz.ai/hands-on-with-feature-engineering-techniques-transforming-variables-acea03472e24

3. Logs Transformation in a Regression Equation
http://www.stat.wharton.upenn.edu/~stine/stat621/handouts/LogsInRegression.pdf

]]> 2020-12-05 23:00:42 UTC https://padlet.com/qiwenlim/f6ov9qemv2dxzb1f/wish/990718157 Contacts limhy2009 https://padlet.com/qiwenlim/f6ov9qemv2dxzb1f/wish/990733354 Temasek Polytechnic  School of Informatics & IT
CDS1C01 Augmented data preparation
AY2020/2021 October semester

Lim Qi Wen       2081769H@student.tp.edu.eg
Lim How Yong   2081766D@student.tp.edu.sg

]]> 2020-12-05 23:22:57 UTC https://padlet.com/qiwenlim/f6ov9qemv2dxzb1f/wish/990733354 one DATAFRAME to rule them all ... data blending limhy2009 https://padlet.com/qiwenlim/f6ov9qemv2dxzb1f/wish/992690843 To facilitate the use of the Airbnb data in the machine learning models (features engineering), data blending was performed to merge the 3 datasets from insideairbnb.com.

Airbnb listings
  • the two listings comprising hosts who registered in different time periods were appended using Data Union
  • both contained the same data headers (i.e., 26 columns of identical datapoints) which were automatically matched using Tibco Spotfire
  • the unified Airbnb listings had 7908 rows vertically stacked in a new data table 

Airbnb neighbourhood
  • the data comprised the "Neighbourhood" column found in Airbnb listing + a categorical column "Region" with labels Central, North, East, West, etc. 
  • Left outer join was used to add the "Region" column to the unified Airbnb listings
  • the joined Airbnb data table had 27 columns
]]> 2020-12-07 03:19:45 UTC https://padlet.com/qiwenlim/f6ov9qemv2dxzb1f/wish/992690843 show me the MONEY ... log-transforming AIRBNB prices limhy2009 https://padlet.com/qiwenlim/f6ov9qemv2dxzb1f/wish/996735871 Predictive modelling involves learning the relationships between a target outcome (dependent variable) and the predictors (independent variables) affecting it. In this case:
  • Target: the price an Airbnb host can charge for her apartment
  • Predictors: the apartment's location, furnishings, other listing attributes, etc.

To improve model accuracies and arrive at the best conclusions, Data Scientists spend much of their time pre-processing data to engineer the best features for use in training their machines.

We attempt to illustrate such a process here, with the price (target) variable; exploring the column in detail and applying log-transformation to handle the outliers found in the column. Tasks performed:

  • Profiling the price column to understand its measure of central tendency (mean, median) and its distribution (skewness, kurtosis)
  • Cleaning empty or missing values, replacing them with the median price
  • Transforming (log10) the cleaned price column in Tibco Spotfire, to re-shape its distribution to mirror a normal distribution approximately.
  • Re-profiling (iterative) the transformed price column to assess its fit for use in modelling.

Applying log-transformation to right-skewed data reduces the distances between data points (from equal to exponential distance) and re-shapes the data distribution to one that is more normally distributed. The Data Science community found the derived regression equations (i.e., in linear regression models) are better-fits of the data throughout the dataset (have smaller RMSEs) and lead to improved predictive powers of the models.

click picture for Tibco Spotfire documentations
]]> 2020-12-08 02:45:29 UTC https://padlet.com/qiwenlim/f6ov9qemv2dxzb1f/wish/996735871 Data profiling - bedrooms qiwenlim https://padlet.com/qiwenlim/f6ov9qemv2dxzb1f/wish/1000724136 The number of bedrooms available in the Airbnb listing for stay will also impact the price to rent out the space at. As such, we will perform data profiling on this column by conducting a semantic check. It is noted that there are 3 empty values found in this column. This would mean that the data for the AirBnB listing may be incomplete. As such, these empty values will be removed. Moreover, when we dive in to the min, max and average, there seems to be two extreme ends i.e 0 and 50 bedrooms which may indicate the presence of an outlier. A bar chart is hence used to analyze the data better and remove the outlier. Upon checking in detail, the outlier is a supposed accommodation for a family of 5 pax. However, it was incorrectly recorded as 50 bedrooms which can influence the pricing of the AirBnB stay depending on the number of pax that can be accommodated. Therefore, this outlier was removed to enhance the quality of the data.]]> 2020-12-09 02:18:20 UTC https://padlet.com/qiwenlim/f6ov9qemv2dxzb1f/wish/1000724136 Data profiling - roomtype qiwenlim https://padlet.com/qiwenlim/f6ov9qemv2dxzb1f/wish/1000975869 In this column, we conduct another profiling for the room type to identify the uniqueness of the data. There are 24 unique count in this column however when we dive in further, there is an outlier which describes the room as Castle. This will impact how the data is used to analyse the relationship between the room type and the price of the AirBnb listing. In order to resolve this, we replaced the Castle to Others so as to narrow down the category for easier analysis.]]> 2020-12-09 04:53:48 UTC https://padlet.com/qiwenlim/f6ov9qemv2dxzb1f/wish/1000975869 Pivoting qiwenlim https://padlet.com/qiwenlim/f6ov9qemv2dxzb1f/wish/1001131392 We further transformed the data from tall and skinny to wide structure so that it is easier to illustrate the analysis of our findings. From this, we found that the regions with the highest average price per room for an Entire Home are most expensive in the Central followed by East, West, North and North East being the least expensive. 

Private rooms are found to be most expensive in Central followed by East, North, North East and West being the least expensive

In light of this, it appears that there are other factors that plays a pivotal role in the price of the AirBnB Listing. The price is not solely dependent on the Region of the property listing but dependent on other factors like the property type and room type to determine the price of the listing.

Typically Central area might be able to fetch a higher price due to its proximity to the city and events / activities going on the area. Other factors such as seasonality, room size, amenities and more can also be taken into consideration which are not present in this data set as it can potentially impact the pricing algorithm in AirBnB]]> 2020-12-09 06:25:15 UTC https://padlet.com/qiwenlim/f6ov9qemv2dxzb1f/wish/1001131392