I am going to demonstrate how to improve speed in R when performing multiple linear regression. Below I compare three methods:
The standard built in R function for regression is lm(). It is the slowest. A bare bones R
implementation is lm.fit() which is substantially faster than lm() but still slow.
The fastest method to perform multiple linear regression is to use Rcpp and RcppArmadillo which is the C++ Armadillo linear algebra library.
A 1253 x 26 design matrix (X) is built from the cars_19 dataset and a simulation is run to compare the three methods:
Multiple linear regression using Rcpp and RcppArmadillo is multiples times faster than the standard R functions!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In previous posts, I used popular machine learning algorithms to fit models to best predict MPG using the cars_19 dataset which is a dataset I created from publicly available data from the Environmental Protection Agency. It was discovered that support vector machine was clearly the winner in predicting MPG and SVM produces models with the lowest RMSE. In this post I am going to use LightGBMto build a predictive model and compare the RMSE to the other models.
Similar to the other models, the variables/features I am using are: Engine displacement (size), number of cylinders, transmission type, number of gears, air inspired method, regenerative braking type, battery capacity Ah, drivetrain, fuel type, cylinder deactivate, and variable valve. The LightGBM package does not handle factors so I will have to transform them into dummy variables. After creating the dummy variables, I will be using 33 input variables.
One of the biggest challenges with this dataset is it is small to be running machine learning models on. The train data set is 939 rows and the test data set is only 314 rows. In an ideal situation there would be more data, but this is real data and all data that is available.
After getting a working model and performing trial and error exploratory analysis to estimate the hyperparameters, I am going to run a grid search using:
As a general rule of thumb num_leaves = 2^(max_depth) and num leaves and max_depth need to be tuned together to prevent overfitting. Solving for max_depth:
max_depth = round(log(num_leaves) / log(2),0)
This is just a guideline, I found values for both hyperparameters higher than the final hyper_grid below caused the model to overfit.
After running a few grid searches, the final hyper_grid I am looking to optimize (minimize RMSE) is 4950 rows. This runs fairly quickly on a Mac mini with the M1 processor and 16 GB RAM making use of the early_stopping_rounds parameter.
postResample(y_test,yhat_predict_final)
RMSE Rsquared MAE
1.7031942 0.9016161 1.2326575
Graph of features that are most explanatory:
75% of the predictions are within 1 standard error of actual, 95% are within 2 standard errors of actual
50% of the predictions are within +- 1 MPG of the actual.
The largest residual in the predicted data is a Hyundai Ioniq which is a very unique data point vs it's peers which explains the RMSE of 1.7 vs MAE of 1.233.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In the previous posts, I used popular machine learning algorithms to fit models to best predict MPG using the cars_19 dataset. It was discovered that support vector machine produced the lowest RMSE. In this post I am going to use XGBoostto build a predictive model and compare the RMSE to the other models.
Similar to the other models, the variables/features I am using are: Engine displacement (size), number of cylinders, transmission type, number of gears, air inspired method, regenerative braking type, battery capacity Ah, drivetrain, fuel type, cylinder deactivate, and variable valve. Unlike the other models, the XGBoost package does not handle factors so I will have to transform them into dummy variables. After creating the dummy variables, I will be using 33 input variables.
After getting a working model and performing trial and error exploratory analysis to estimate the eta and tree depth hyperparameters, I am going to run a grid search. I am going to run 64 XGBoost models
Comparison of RMSE:
svm = .93
XGBoost = 1.74
gradient boosting = 1.8
random forest = 1.9
neural network = 2.06
decision tree = 2.49
mlr = 2.6
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
I built robustreg in 2006 and at the time the major stat packages did not have a robust regression available. Below are graphs of weekly and cumulative downloads from just the RStudio mirror. I would estimate total downloads at over 150,000.
The median_rcpp() function is written in C++ and is multiple times faster than the R base function median().
> r_norm<- rnorm(1000000)
> system.time(median(r_norm))
user system elapsed
0.040 0.004 0.044
> system.time(median_rcpp(r_norm))
user system elapsed
0.011 0.000 0.011
In the previous post I fitted a neural network to the cars_19 dataset using the neuralnet package. In this post I am going to use TensorFlow to fit a deep neural network using the same data.
The main difference between the neuralnet package and TensorFlow is TensorFlow uses the adagrad optimizer by default whereas neuralnet uses rprop+ Adagrad is a modified stochastic gradient descent optimizer with a per-parameter learning rate.
The data which is all 2019 vehicles which are non pure electric (1253 vehicles) are summarized in previous posts below.
To prepare the data to fit the neural network, TensorFlow requires categorical variables to be converted into a dense representation by using the column_embedding() function.
Similar to the neural network I fitted using neuralnet(), I am going to use two hidden layers with seven and three neurons respectively.
Train, evaluate, and predict:
#Create a deep neural network (DNN) estimator.
model <- dnn_regressor(hidden_units=c(7,3),feature_columns = cols)
set.seed(123)
indices <- sample(1:nrow(cars_19), size = 0.75 * nrow(cars_19))
train <- cars_19[indices, ]
test <- cars_19[-indices, ]
#train model
model %>% train(cars_19_input_fn(train, num_epochs = 1000))
#evaluate model
model %>% evaluate(cars_19_input_fn(test))
#predict
yhat <- model %>% predict(cars_19_input_fn(test))
yhat <- unlist(yhat)
y <- test$fuel_economy_combined
postResample(yhat, y)
RMSE Rsquared MAE
1.9640173 0.8700275 1.4838347
The results are similar to the other models and neuralnet().
I am going to look at the error rate in TensorBoard which is a visualization tool. TensorBoard is great for visualizing TensorFlow graphs and for plotting quantitative metrics about the execution of the
graph. Below is the mean squared error at each iteration. It stabilizes fairly quickly. Next post I will get into TensorBoard in a lot more depth.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In the previous four posts I have used multiple linear regression, decision trees, random forest, gradient boosting, and support vector machine to predict MPG for 2019 vehicles. It was determined that svm produced the best model. In this post I am going to use the neuralnet package to fit a neural network to the cars_19 dataset.
Similar to the other models, the variables/features I am using are: Engine displacement (size), number of cylinders, transmission type, number of gears, air inspired method, regenerative braking type, battery capacity Ah, drivetrain, fuel type, cylinder deactivate, and variable valve. Unlike the other models, the neuralnet package does not handle factors so I will have to transform them into dummy variables. After creating the dummy variables, I will be using 33 input variables.
The data which is all 2019 vehicles which are non pure electric (1253 vehicles) are summarized in previous posts below.
Neuralnet will not accept a formula like fuel_economy_combined ~. so I need to write out the full model.
n <- names(cars_19)
f <- as.formula(paste("fuel_economy_combined ~", paste(n[!n %in% "fuel_economy_combined"], collapse = " + ")))
Next I need to transform all of the factor variables into binary dummy variables.
m <- model.matrix(f, data = tmp)
m <- as.matrix(data.frame(m, tmp[, 1]))
colnames(m)[28] <- "fuel_economy_combined")
I am going to use the geometric pyramid rule to determine the amount of hidden layers and neurons for each layer. The general rule of thumb is if the data is linearly separable, use one hidden layer and if it is non-linear use two hidden layers. I am going to use two hidden layers as I already know the non-linear svm produced the best model.
r = (INP_num/OUT_num)^(1/3)
HID1_num = OUT_num*(r^2) #number of neurons in the first hidden layer
HID2_num = OUT_num*r #number of neurons in the second hidden layer
This suggests I use two hidden layers with 9 neurons in the first layer and 3 neurons in the second layer. I originally fit the model with this combination but it turned out to overfit. As this is just a suggestion, I found that two hidden layers with 7 and 3 neurons respectively produced the best neural network that did not overfit.
postResample(yhat, y)
RMSE Rsquared MAE
2.0036294 0.8688363 1.4894264
I am going to run a 20 fold cross validation to estimate error better as these results are dependent on sample and initialization of the neural network.
Comparison of RMSE:
svm = .93
gradient boosting = 1.8
random forest = 1.9
neural network = 2.06
decision tree = 2.49
mlr = 2.6
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
In the previous three posts I used multiple linear regression, decision trees, gradient boosting, and support vector machine to predict miles per gallon for 2019 vehicles. It was determined that svm produced the best model. In this post, I am going to run TensorFlow through R and fit a multiple linear regression model using the same data to predict MPG.
There are 1253 vehicles in the cars_19 dataset. I am simply running mlr using Tensorflow for demonstrative purposes as using lm() in R is more efficient and more precise for such a small dataset.
TensorFlow uses an algorithm that is dependent upon convergence whereas R computes the closed form estimates of beta. I will be using 11 features and an intercept so R will be inverting a 12 x 12 matrix which is not computationally expensive with today's technology.
The dataset below of 11 features contains 7 factor variables and 4 numeric variables.
#input_fn for a given subset of data
cars_19_input_fn <- function(data, num_epochs = 1) {
input_fn(
data,
features = colnames(cars_19[c(2:12)]),
response = "fuel_economy_combined",
batch_size = 64,
num_epochs = num_epochs
)
}
Train, evaluate, predict:
model <- linear_regressor(feature_columns = cols)
set.seed(123)
indices <- sample(1:nrow(cars_19), size = 0.75 * nrow(cars_19))
train <- cars_19[indices, ]
test <- cars_19[-indices, ]
#train model
model %>% train(cars_19_input_fn(train, num_epochs = 1000))
#evaluate model
model %>% evaluate(cars_19_input_fn(test))
#predict
yhat <- model %>% predict(cars_19_input_fn(test))
Results are very close to the R closed form estimates:
postResample(yhat, y)
RMSE Rsquared MAE
2.5583185 0.7891934 1.9381757
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The variables/features I am using for the models are: Engine displacement (size), number of cylinders, transmission type, number of gears, air inspired method, regenerative braking type, battery capacity Ah, drivetrain, fuel type, cylinder deactivate, and variable valve.
There are 1253 vehicles in the dataset (does not include pure electric vehicles) summarized below.
fuel_economy_combined eng_disp num_cyl transmission
Min. :11.00 Min. :1.000 Min. : 3.000 A :301
1st Qu.:19.00 1st Qu.:2.000 1st Qu.: 4.000 AM : 46
Median :23.00 Median :3.000 Median : 6.000 AMS: 87
Mean :23.32 Mean :3.063 Mean : 5.533 CVT: 50
3rd Qu.:26.00 3rd Qu.:3.600 3rd Qu.: 6.000 M :148
Max. :58.00 Max. :8.000 Max. :16.000 SA :555
SCV: 66
num_gears air_aspired_method
Min. : 1.000 Naturally Aspirated :523
1st Qu.: 6.000 Other : 5
Median : 7.000 Supercharged : 55
Mean : 7.111 Turbocharged :663
3rd Qu.: 8.000 Turbocharged+Supercharged: 7
Max. :10.000
regen_brake batt_capacity_ah
No :1194 Min. : 0.0000
Electrical Regen Brake: 57 1st Qu.: 0.0000
Hydraulic Regen Brake : 2 Median : 0.0000
Mean : 0.3618
3rd Qu.: 0.0000
Max. :20.0000
drive cyl_deactivate
2-Wheel Drive, Front :345 Y: 172
2-Wheel Drive, Rear :345 N:1081
4-Wheel Drive :174
All Wheel Drive :349
Part-time 4-Wheel Drive: 40
fuel_type
Diesel, ultra low sulfur (15 ppm, maximum): 28
Gasoline (Mid Grade Unleaded Recommended) : 16
Gasoline (Premium Unleaded Recommended) :298
Gasoline (Premium Unleaded Required) :320
Gasoline (Regular Unleaded Recommended) :591
variable_valve
N: 38
Y:1215
svm_stats
RMSE Rsquared MAE
0.9331948 0.9712492 0.7133039
The tuned support vector machine outperforms the gradient boosted model substantially with a MSE of .87 vs a MSE of 3.25 for the gradient boosted model and a MSE of 3.67 for the random forest.
summary(m_svm_tuned)
Call:
svm(formula = fuel_economy_combined ~ ., data = test, gamma = 0.25, cost = 32, scale = TRUE)
Parameters:
SVM-Type: eps-regression
SVM-Kernel: radial
cost: 32
gamma: 0.25
epsilon: 0.1
Number of Support Vectors: 232
sum(abs(res)<=1) / 314
[1] 0.8503185
The model is able to predict 85% of vehicles within 1 MPG of EPA estimate. Considering I am not rounding this is a great result.
The model also does a much better job with outliers as none of the models predicted the Hyundai Ioniq well.
tmp[which(abs(res) > svm_stats[1] * 3), ] #what cars are 3 se residuals
Division Carline fuel_economy_combined pred_svm_tuned
641 HYUNDAI MOTOR COMPANY Ioniq 55 49.01012
568 TOYOTA CAMRY XSE 26 22.53976
692 Volkswagen Arteon 4Motion 23 26.45806
984 Volkswagen Atlas 19 22.23552
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The variables/features I am using for the models are: Engine displacement (size), number of cylinders, transmission type, number of gears, air inspired method, regenerative braking type, battery capacity Ah, drivetrain, fuel type, cylinder deactivate, and variable valve.
There are 1253 vehicles in the dataset (does not include pure electric vehicles) summarized below.
fuel_economy_combined eng_disp num_cyl transmission
Min. :11.00 Min. :1.000 Min. : 3.000 A :301
1st Qu.:19.00 1st Qu.:2.000 1st Qu.: 4.000 AM : 46
Median :23.00 Median :3.000 Median : 6.000 AMS: 87
Mean :23.32 Mean :3.063 Mean : 5.533 CVT: 50
3rd Qu.:26.00 3rd Qu.:3.600 3rd Qu.: 6.000 M :148
Max. :58.00 Max. :8.000 Max. :16.000 SA :555
SCV: 66
num_gears air_aspired_method
Min. : 1.000 Naturally Aspirated :523
1st Qu.: 6.000 Other : 5
Median : 7.000 Supercharged : 55
Mean : 7.111 Turbocharged :663
3rd Qu.: 8.000 Turbocharged+Supercharged: 7
Max. :10.000
regen_brake batt_capacity_ah
No :1194 Min. : 0.0000
Electrical Regen Brake: 57 1st Qu.: 0.0000
Hydraulic Regen Brake : 2 Median : 0.0000
Mean : 0.3618
3rd Qu.: 0.0000
Max. :20.0000
drive cyl_deactivate
2-Wheel Drive, Front :345 Y: 172
2-Wheel Drive, Rear :345 N:1081
4-Wheel Drive :174
All Wheel Drive :349
Part-time 4-Wheel Drive: 40
fuel_type
Diesel, ultra low sulfur (15 ppm, maximum): 28
Gasoline (Mid Grade Unleaded Recommended) : 16
Gasoline (Premium Unleaded Recommended) :298
Gasoline (Premium Unleaded Required) :320
Gasoline (Regular Unleaded Recommended) :591
variable_valve
N: 38
Y:1215
Starting with an untuned base model:
trees <- 1200
m_boosted_reg_untuned <- gbm(
formula = fuel_economy_combined ~ .,
data = train,
n.trees = trees,
distribution = "gaussian"
)
After running the grid search, it is apparent that there is overfitting. This is something to be very careful about. I am going to run a 5 fold cross validation to estimate out of bag error vs MSE. After running the 5 fold CV, this is the best model that does not overfit:
The tuned gradient boosted model performs better than the random forest with a MSE of 3.25 vs 3.67 for the random forest.
> summary(res)
Min. 1st Qu. Median Mean 3rd Qu. Max.
-5.40000 -0.90000 0.00000 0.07643 1.10000 9.10000
50% of the predictions are within 1 MPG of the EPA Government Estimate.
The largest residuals are exotics and a hybrid which are the more unique data points in the dataset.
> tmp[which(abs(res) > boosted_stats[1] * 3), ]
Division Carline fuel_economy_combined pred_boosted_reg
642 HYUNDAI MOTOR COMPANY Ioniq Blue 58 48.5
482 KIA MOTORS CORPORATION Forte FE 35 28.7
39 Lamborghini Aventador Coupe 11 17.2
40 Lamborghini Aventador Roadster 11 17.2
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters