Dataset Exploration and Analysis #2

The cover image is generated by ChatGPT and it’s about dataset exploration and analysis. The prompt used was: “A digital illustration in a 16:9 aspect ratio, showing a young male data analyst working in a modern high-tech office, analyzing a large dataset on a widescreen monitor. The screen displays various data visualizations: bar charts, scatter plots, histograms, and data tables. The interface is sleek and futuristic, with turquoise and blue tones, glowing UI elements, and a clear emphasis on Dataset Exploration and Analysis” 。

Introduction

The dataset used in this project is sourced from Kaggle and was provided by Adil Shamim. It originates from the UCI Machine Learning Repository and was initially introduced by P. Cortez and A. Silva in their study titled Using Data Mining to Predict Secondary School Student Performance. This dataset is part of research presented at the 5th FUBUTEC 2008 conference held in Porto, Portugal. The variables G1, G2, G3, and absences were directly obtained from the school records, while the other variables were collected through questionnaires.

In this analysis, Python will be used as the programming language, and the main objective is to identify the variables that influence the final semester grade G3.

Dataset

Variable Description

The dataset contains 399 records and 33 variables. The variables are described as follows:

Dataset

The following analysis is conducted using Python as the programming language. The original contents of the dataset are as follows:

Original Data Bar Charts

To better prepare for data processing, we begin by plotting bar charts for various variables to gain a basic understanding of the dataset.

From the charts, we can observe the following:

1. The age distribution shows slight variations, with most students aged between 15 and 18. This suggests that the dataset mainly consists of high school students, especially by Taiwanese standards.
2. There are students aged 20, 21, and 22, which may indicate that some are repeating a grade or have special circumstances.
3. The lowest category of studytime is around 2 hours.
4. The distributions of freetime, G1, G2, and G3 are approximately normal.
5. absences, Dalc, and Walc are right-skewed.
6. famrel and health show a left-skewed distribution.
7. Parents’ occupations are most commonly labeled as other and services.
8. The gender ratio is roughly balanced.

Data Cleaning

To ensure a smooth analysis process, we begin by organizing the categorical variables, assigning them as either factors or using one-hot encoding. Additionally, we group related variables and specify G3 as the target variable for prediction.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

## factor
factors = {
    'schoolsup': ['no', 'yes'],
    'famsup': ['no', 'yes'],
    'paid': ['no', 'yes'],
    'activities': ['no', 'yes'],
    'nursery': ['no', 'yes'],
    'higher': ['no', 'yes'],
    'internet': ['no', 'yes'],
    'romantic': ['no', 'yes']
}

## one-hot encoding
one_hot_encoding = {
    'school': ['GP', 'MS'],
    'sex': ['F', 'M'],
    'address': ['U', 'R'],
    'famsize': ['GT3', 'LE3'],
    'Pstatus': ['T', 'A'],
    'Mjob': ['other', 'at_home', 'health', 'services', 'teacher'],
    'Fjob': ['other', 'at_home', 'health', 'services', 'teacher'],
    'reason': ['other', 'home', 'course', 'reputation'],
    'guardian': ['other', 'mother', 'father']
}

To facilitate data processing, we first organize the categorical variables. Those with an inherent order are converted into factors with levels arranged from weakest to strongest. Variables without a natural order are handled using one-hot encoding, and we adjust the order of categories within each variable for consistency, making future visual inspection more convenient.

From the previous bar charts, we observed that the school MS had significantly fewer data entries. To avoid biased analysis results caused by data imbalance between schools, we choose to exclude students from the MS school and focus our analysis solely on the GP school.

Additionally, G1, G2, and G3 all represent summative assessments—evaluations of students’ academic performance at different stages of the semester. We will later visualize the correlation among these three grades to demonstrate their strong interrelationship.

Since our goal is to identify the factors that influence final academic performance, we save a copy of the original dataset, remove G1 and G2, and retain G3 as the target variable for prediction.

Bar Charts After Data Cleaning

The following bar charts illustrate the dataset after the cleaning process. After removing data entries from the MS school, the overall structure remains largely the same, though some differences compared to the original dataset are noticeable.

Additionally, some variables have been converted from categorical to numerical form during preprocessing, which is reflected in the charts as numeric values.

These updated bar charts now provide a clearer and more focused view of the information specific to the GP school.

Correlation Matrix and Scatter Plot

We can examine the relationships between variables by calculating the correlation matrix, which quantifies pairwise correlations. A more advanced approach is to visualize this matrix using a correlation heatmap. In the plot below, deeper red colors indicate stronger positive correlations**, while deeper blue colors indicate stronger negative correlations.

As shown in the figure, there is a very strong correlation among G1, G2, and G3. This is precisely why we decided to remove G1 and G2 from the dataset. If they were kept, analysis results would overwhelmingly reflect their influence on G3, overshadowing the impact of other variables.

Below is a 3D scatter plot of G1, G2, and G3, where we can observe a strong linear relationship among these three variables.

If you are unable to view the interactive 3D scatter plot or need to view it in full screen, please click here.

Next, we present the correlation plots within other variable groups. From the following figures, we can observe:

• Within the parents group, both Medu, Fedu, Mjob and Fjob show strong positive correlations, indicating that the parents group can be used to analyze the influence of parents on student academic performance.
• Within the alcohol group, Dalc and Walc are highly positively correlated, but they show almost no correlation with the students’ self-assessed health status.
• Within the support group, there is a positive correlation between famsup and paid, which may suggest that family influences participation in extracurricular courses.
• Within the after_class group, freetime and goout are positively correlated, possibly indicating that students often spend their free time socializing with friends after school.

Interaction Bar Charts of Variables

Below, we select and plot some bar charts showing interactions between variables to observe their data patterns.

First, we plot the average minimum weekly study time (studytime) by age. The studytime variable is coded as follows:

• 1 means less than 2 hours per week
• 2 means 2 to 5 hours per week
• 3 means 5 to 10 hours per week
• 4 means more than 10 hours per week

From this, we know the minimum weekly study time for each code is:

• 1 at least 0 hours per week
• 2 at least 2 hours per week
• 3 at least 5 hours per week
• 4 at least 10 hours per week

Therefore, we can draw the following bar chart showing each age group’s average minimum weekly study time. From the chart, we find that students aged 15 to 19 study at least 2 to 3 hours per week on average, with 18-year-olds averaging even 3 hours or more. We speculate that students aged 18 to 19 might have more study pressure due to college entrance exams, hence the longer study times. Meanwhile, students aged 20 and above are likely less interested in studying or may have other career plans.

Next, we plot the bar charts of after-school free time (freetime) and frequency of going out with friends (goout). We observe that students aged 15 to 19 have average scores around 3 for both free time and going out frequency, while students aged 20 and above tend to have even higher averages than the younger group. Comparing with the previous chart, it’s clear that students aged 20 and above differ significantly from those aged 15 to 19, so we should pay close attention to the effect of the age variable in later analyses.

Next is the bar chart showing students’ willingness to pursue higher education (higher) and their average final grade G3. From this chart, we can see that students who intend to continue their studies tend to have higher average G3 scores than those who do not, indicating that the intention to pursue further education is one factor influencing academic performance.

The following chart is particularly interesting. According to Portuguese law, the legal age of adulthood and the legal age for purchasing and consuming alcohol is 18. In the original dataset questionnaire, the lowest score for drinking behavior is 1, which represents very little or no alcohol consumption. Thus, students who do not drink at all should mostly score 1.

In the chart below showing health status and average weekday and weekend alcohol consumption by age, we visualize students’ self-assessed health along with their average alcohol consumption on weekdays and weekends. To facilitate intuitive comparison between weekly drinking amounts and health status, we stacked weekday and weekend alcohol consumption, assigning 4 days for weekdays and 3 days for weekends. This reflects that students spend most daytime hours at school (weekdays) and tend to drink mostly in the evenings and weekends, with Friday night often regarded as the start of the weekend.

Notably, some students under 18 show average alcohol consumption scores above 1, implying that some minors in the dataset do consume alcohol.

We want to find the top 10 most frequent combinations of Medu, Fedu, Mjob, Fjob, and compute the average G3 for each combination.

In both plots below, the blue bars on the left show the number of times each combination appears, and the red bars on the right show the average G3 for each combination.

After we computed the average G3 for the top 10 most frequent combinations, we want to find the top 10 combinations with the highest average G3.

We can see that when both Medu and Fedu are higher, and both Mjob and Fjob are teachers, the students’ average G3 is 13.00; when Fjob is other, the average G3 is 9.00; and when Mjob is services, the average G3 is 15.20.

Therefore, we can see that, at the same education levels, different Mjob and Fjob combinations also affect G3.

Multidimensional Scaling

The original dataset contains 33 variables, most of which have very low correlations with each other (close to zero), making it difficult to directly identify which variables relate to the G3 final grade. To address the problem of having too many variables with diluted explanatory power, we first grouped the variables based on their attributes and meanings. Then, for each group, we applied Multidimensional Scaling (MDS) to perform dimensionality reduction.

During the MDS process, we compared the distances among student samples within each group, compressing the originally high-dimensional variables into two dimensions. This approach attempts to preserve the relative relationships among samples as much as possible, reducing the number of variables while improving each group’s explanatory power regarding G3. The two new variables generated after MDS for each group are prefixed with the group name and named _dim1 and _dim2 respectively.

Below are the visualizations of the MDS results for each group, with student samples colored according to their corresponding G3 scores, to visually present the potential relationships between the reduced dimensions and academic performance.

Although we do not analyze G1, G2, and G3 directly, we still plot the samples of these variables after MDS dimensionality reduction for comparison with the other groups above.

After applying MDS, we then plot the correlation matrix to examine the relationships between the reduced variables.

If you are unable to view the interactive correlation matrix visualization or need to view it in full screen, please click here.

Pair Plot

Below is a pair plot to observe if the variables after MDS show stronger correlations.

Principal Component Analysis

Before performing Principal Component Analysis (PCA), we first standardize the variables. After standardizing all variables except the target variable G3, the data appears as follows:

The most important feature is selected as the variable with the largest absolute loading value within each principal component.

Plotting the variance helps us better analyze the dataset. The black line represents the explained variance of each principal component, the blue line shows the remaining unexplained variance, and the red line indicates the cumulative explained variance.

Of course, we can also calculate and present all variables transformed by PCA as shown below.

The following shows a 3D scatter plot of the loadings for each variable on PC1, PC2, and PC3, along with their cumulative explained variance.

If you are unable to view the interactive PC1 vs. PC2 vs. PC3 3D scatter plot or need to view it in full screen, please click here.

Regression Model

Python offers many modules for regression modeling, such as sklearn and statsmodels. Among them:

• sklearn is more oriented towards machine learning, emphasizing prediction and generalization ability, but many statistical metrics need to be coded manually.
• statsmodels focuses more on statistical modeling and inference, making it more suitable for academic or statistical analysis, and it provides convenient model summaries.

Below, we use the statsmodels module for analysis and plotting.

Original Dataset

We first run regression on the cleaned original dataset and observe:

• The original model has an $R^2$ of 0.269, indicating relatively low explanatory power. The adjusted $R^2$ is even lower, suggesting some variables may be redundant.
• The overall model P-value = $1.80 \times 10^{-9}$, indicating that some variables have a significant effect on G3.
  • failures has a P-value of 0.000 and a negative coefficient, meaning the number of failures has a significant negative effect on G3; more failures lead to lower grades.
  • goout has a P-value of 0.001 and a negative coefficient, indicating that frequency of going out has a significant negative effect on G3.
  • schoolsup has a P-value of 0.000 and a negative coefficient, showing that students receiving extra school support tend to have lower grades, possibly because they need help due to learning difficulties.
  • romantic has a P-value of 0.019 and a negative coefficient, indicating that having romantic experience has a significant negative effect on G3.
  • sex_M has a P-value of 0.015 and a positive coefficient, meaning male students tend to have significantly higher grades than females.
  • address_0 has a P-value of 0.005 and a positive coefficient, indicating students living in urban areas score higher.
  • famsize_1 has a P-value of 0.004 and a positive coefficient, suggesting students from larger families perform significantly better.
  • Pstatus_1 has a P-value of 0.020 and a positive coefficient, indicating students with separated parents have significantly higher grades.
  • Mjob_2 (mother is a healthcare worker) has a P-value of 0.047 and a positive coefficient, showing this occupation positively affects student grades.
  • Mjob_3 (mother is a civil servant) has a P-value of 0.012 and a positive coefficient, indicating a significant positive effect on grades from this occupation.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72

                             OLS Regression Results                            
==============================================================================
Dep. Variable:                     G3   R-squared:                       0.269
Model:                            OLS   Adj. R-squared:                  0.195
Method:                 Least Squares   F-statistic:                     3.634
Date:                Fri, 13 Jun 2025   Prob (F-statistic):           1.80e-09
Time:                        17:00:54   Log-Likelihood:                -974.55
No. Observations:                 349   AIC:                             2015.
Df Residuals:                     316   BIC:                             2142.
Df Model:                          32                                         
Covariance Type:            nonrobust                                         
==============================================================================
                 coef    std err          t      P>|t|      [0.025      0.975]
------------------------------------------------------------------------------
age           -0.2420      0.215     -1.123      0.262      -0.666       0.182
Medu           0.2619      0.353      0.742      0.459      -0.433       0.957
Fedu          -0.2358      0.300     -0.786      0.432      -0.826       0.355
traveltime    -0.3604      0.374     -0.964      0.336      -1.096       0.375
studytime      0.4272      0.305      1.403      0.162      -0.172       1.027
failures      -1.9093      0.345     -5.538      0.000      -2.588      -1.231
famrel         0.3023      0.272      1.110      0.268      -0.233       0.838
freetime       0.3111      0.255      1.219      0.224      -0.191       0.813
goout         -0.8290      0.241     -3.437      0.001      -1.303      -0.354
Dalc          -0.4425      0.368     -1.204      0.230      -1.166       0.281
Walc           0.4851      0.262      1.854      0.065      -0.030       1.000
health        -0.1634      0.175     -0.932      0.352      -0.508       0.182
absences       0.0537      0.030      1.791      0.074      -0.005       0.113
schoolsup     -1.7586      0.489     -3.598      0.000      -2.720      -0.797
famsup        -1.7586      0.489     -3.598      0.000      -2.720      -0.797
paid          -1.7586      0.489     -3.598      0.000      -2.720      -0.797
activities    -1.7586      0.489     -3.598      0.000      -2.720      -0.797
nursery       -1.7586      0.489     -3.598      0.000      -2.720      -0.797
higher        -1.7586      0.489     -3.598      0.000      -2.720      -0.797
internet       0.4218      0.694      0.608      0.544      -0.943       1.786
romantic      -1.1990      0.509     -2.354      0.019      -2.201      -0.197
sex_M          1.2575      0.514      2.445      0.015       0.246       2.269
address_0      1.0845      0.386      2.813      0.005       0.326       1.843
address_1      0.6741      0.427      1.579      0.115      -0.166       1.514
famsize_0      0.7028      0.358      1.965      0.050      -0.001       1.406
famsize_1      1.0557      0.359      2.937      0.004       0.348       1.763
Pstatus_0      0.6980      0.448      1.557      0.120      -0.184       1.580
Pstatus_1      1.0606      0.453      2.343      0.020       0.170       1.951
Mjob_0         0.0788      0.442      0.178      0.859      -0.791       0.948
Mjob_1        -0.4112      0.688     -0.598      0.550      -1.765       0.942
Mjob_2         1.4522      0.729      1.992      0.047       0.018       2.887
Mjob_3         1.1294      0.448      2.518      0.012       0.247       2.012
Mjob_4        -0.4907      0.654     -0.750      0.454      -1.778       0.796
Fjob_0        -0.3711      0.481     -0.772      0.441      -1.317       0.575
Fjob_1         0.1896      0.944      0.201      0.841      -1.667       2.046
Fjob_2         0.6282      0.896      0.701      0.484      -1.135       2.392
Fjob_3        -0.2106      0.513     -0.410      0.682      -1.220       0.799
Fjob_4         1.5225      0.789      1.929      0.055      -0.030       3.075
reason_0       1.0306      0.678      1.520      0.129      -0.303       2.364
reason_1       0.1274      0.432      0.295      0.768      -0.723       0.977
reason_2      -0.0459      0.434     -0.106      0.916      -0.899       0.807
reason_3       0.6465      0.446      1.450      0.148      -0.231       1.524
guardian_0     0.7979      0.724      1.103      0.271      -0.626       2.221
guardian_1     0.6490      0.383      1.695      0.091      -0.104       1.402
guardian_2     0.3117      0.466      0.668      0.505      -0.606       1.229
==============================================================================
Omnibus:                       20.903   Durbin-Watson:                   2.172
Prob(Omnibus):                  0.000   Jarque-Bera (JB):               23.006
Skew:                          -0.618   Prob(JB):                     1.01e-05
Kurtosis:                       3.234   Cond. No.                     1.22e+16
==============================================================================

Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
[2] The smallest eigenvalue is 9.56e-28. This might indicate that there are
strong multicollinearity problems or that the design matrix is singular.
MSE:
 62.58324935862237

Dataset After MDS Transformation

• The overall model P-value = $4.23 \times 10^{-8}$, indicating the model’s prediction of G3 is statistically significant.
  • age has a P-value of 0.011 with a negative coefficient, meaning that as age increases, the grades tend to decrease.
  • support_dim1 has a P-value of 0.009 with a positive coefficient, indicating that the first dimension of support-related features (such as school and family support) is significantly positively correlated with G3.
  • support_dim2 has a P-value of 0.005 with a positive coefficient, showing that the second dimension of support also has a significant positive effect.
  • family_dim2 has a P-value of 0.009 with a positive coefficient, meaning the second dimension of family-related factors is significantly positively associated with G3.
  • parents_dim1 has a P-value of 0.046 with a negative coefficient, suggesting some parental background factors may have a significant negative impact on G3.
  • performance_dim1 has a P-value of 0.030 with a negative coefficient, possibly indicating some academic performance background factors negatively relate to G3.
  • after_class_dim1 has a P-value of 0.014 with a positive coefficient, indicating the first dimension of after-school activities has a significant positive influence on G3.
  • after_class_dim2 has a P-value of 0.034 with a positive coefficient, showing the second dimension of after-school activities also has a significant positive effect.
  • romantic has a P-value of 0.008 with a negative coefficient, indicating that having a romantic relationship significantly negatively affects G3.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

                             OLS Regression Results                            
==============================================================================
Dep. Variable:                     G3   R-squared:                       0.188
Model:                            OLS   Adj. R-squared:                  0.143
Method:                 Least Squares   F-statistic:                     4.232
Date:                Fri, 13 Jun 2025   Prob (F-statistic):           4.23e-08
Time:                        17:00:54   Log-Likelihood:                -992.98
No. Observations:                 349   AIC:                             2024.
Df Residuals:                     330   BIC:                             2097.
Df Model:                          18                                         
Covariance Type:            nonrobust                                         
======================================================================================
                         coef    std err          t      P>|t|      [0.025      0.975]
--------------------------------------------------------------------------------------
const                 19.2013      3.565      5.386      0.000      12.189      26.214
age                   -0.5347      0.210     -2.552      0.011      -0.947      -0.122
support_dim1           1.1434      0.434      2.633      0.009       0.289       1.998
support_dim2           1.3208      0.471      2.804      0.005       0.394       2.247
family_dim1            0.1546      0.254      0.608      0.544      -0.346       0.655
family_dim2            0.6712      0.254      2.643      0.009       0.172       1.171
parents_dim1          -0.2608      0.130     -2.003      0.046      -0.517      -0.005
parents_dim2          -0.0731      0.141     -0.518      0.605      -0.351       0.205
performance_dim1      -0.2394      0.110     -2.175      0.030      -0.456      -0.023
performance_dim2       0.0689      0.069      0.996      0.320      -0.067       0.205
alcohol_dim1          -0.1183      0.169     -0.701      0.484      -0.450       0.213
alcohol_dim2           0.1784      0.184      0.971      0.332      -0.183       0.540
after_class_dim1       0.5180      0.211      2.460      0.014       0.104       0.932
after_class_dim2       0.5403      0.254      2.129      0.034       0.041       1.039
school_choice_dim1     0.7140      0.468      1.526      0.128      -0.206       1.634
school_choice_dim2     0.5168      0.492      1.050      0.295      -0.452       1.485
internet               0.2191      0.685      0.320      0.749      -1.129       1.567
romantic              -1.3662      0.513     -2.665      0.008      -2.375      -0.358
sex_M                  0.7956      0.520      1.531      0.127      -0.227       1.818
==============================================================================
Omnibus:                       20.982   Durbin-Watson:                   2.109
Prob(Omnibus):                  0.000   Jarque-Bera (JB):               23.167
Skew:                          -0.623   Prob(JB):                     9.32e-06
Kurtosis:                       3.200   Cond. No.                         260.
==============================================================================

Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
MSE:
 77.57679234040314

Dataset After MDS and PCA Transformation

All data points were transformed using PCA, and then regression analysis was performed. From the regression model, we observed:

• The MSE of the PCA-transformed data is the same as that of the previous model.
• When performing regression using the principal components, we found that PC1, PC4, PC9, PC10, and PC15 have a more significant impact on G3.

From the earlier analysis of important variables within each principal component, we found that the variables influencing this model are:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

                             OLS Regression Results                            
==============================================================================
Dep. Variable:                     G3   R-squared:                       0.188
Model:                            OLS   Adj. R-squared:                  0.143
Method:                 Least Squares   F-statistic:                     4.232
Date:                Fri, 13 Jun 2025   Prob (F-statistic):           4.23e-08
Time:                        17:00:54   Log-Likelihood:                -992.98
No. Observations:                 349   AIC:                             2024.
Df Residuals:                     330   BIC:                             2097.
Df Model:                          18                                         
Covariance Type:            nonrobust                                         
==============================================================================
                 coef    std err          t      P>|t|      [0.025      0.975]
------------------------------------------------------------------------------
const         19.2013      3.565      5.386      0.000      12.189      26.214
PC1            0.4910      0.167      2.947      0.003       0.163       0.819
PC2           -0.3802      0.242     -1.574      0.117      -0.856       0.095
PC3            0.1271      0.252      0.504      0.614      -0.368       0.623
PC4           -0.7924      0.341     -2.323      0.021      -1.464      -0.121
PC5            0.3656      0.230      1.590      0.113      -0.087       0.818
PC6            0.2171      0.321      0.677      0.499      -0.414       0.848
PC7            0.4347      0.292      1.486      0.138      -0.141       1.010
PC8           -0.4930      0.352     -1.402      0.162      -1.185       0.199
PC9           -1.5652      0.382     -4.092      0.000      -2.318      -0.813
PC10           1.2666      0.438      2.893      0.004       0.405       2.128
PC11          -0.1674      0.302     -0.555      0.579      -0.761       0.426
PC12           0.1927      0.319      0.603      0.547      -0.436       0.821
PC13           0.4138      0.506      0.818      0.414      -0.581       1.409
PC14           0.3158      0.287      1.099      0.273      -0.249       0.881
PC15          -1.0856      0.463     -2.346      0.020      -1.996      -0.175
PC16          -0.2603      0.375     -0.694      0.488      -0.998       0.478
PC17           0.6920      0.621      1.115      0.266      -0.529       1.913
PC18           0.3474      0.179      1.942      0.053      -0.005       0.699
==============================================================================
Omnibus:                       20.982   Durbin-Watson:                   2.109
Prob(Omnibus):                  0.000   Jarque-Bera (JB):               23.167
Skew:                          -0.623   Prob(JB):                     9.32e-06
Kurtosis:                       3.200   Cond. No.                         260.
==============================================================================

Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
MSE:
 77.57679234040309

Dataset After MDS and PCA Transformation Using 80% Explained Variance

We conducted regression analysis using only the principal components that collectively explain 80% of the total variance. The results show:

• The $R^2$ of the model using the top 80% explained variance PCs is 0.137, indicating a decrease in explanatory power compared to the original dataset model.
• From the regression results, we found that PC1, PC4, PC5, and PC9 have a more significant impact on G3.

Since this dataset can still be directly fed into a regression model for training, there is no need to apply PCA for dimensionality reduction beforehand. Doing so would actually reduce the model’s explanatory power and increase the mean squared error (MSE).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

                             OLS Regression Results                            
==============================================================================
Dep. Variable:                     G3   R-squared:                       0.137
Model:                            OLS   Adj. R-squared:                  0.109
Method:                 Least Squares   F-statistic:                     4.861
Date:                Fri, 13 Jun 2025   Prob (F-statistic):           5.57e-07
Time:                        17:00:54   Log-Likelihood:                -1003.5
No. Observations:                 349   AIC:                             2031.
Df Residuals:                     337   BIC:                             2077.
Df Model:                          11                                         
Covariance Type:            nonrobust                                         
==============================================================================
                 coef    std err          t      P>|t|      [0.025      0.975]
------------------------------------------------------------------------------
const         14.5323      2.774      5.238      0.000       9.075      19.990
PC1            0.4738      0.145      3.264      0.001       0.188       0.759
PC2           -0.3475      0.212     -1.641      0.102      -0.764       0.069
PC3            0.0787      0.230      0.342      0.733      -0.374       0.531
PC4           -0.5873      0.200     -2.935      0.004      -0.981      -0.194
PC5            0.4230      0.212      1.993      0.047       0.005       0.841
PC6           -0.1392      0.263     -0.528      0.598      -0.657       0.379
PC7            0.3270      0.251      1.301      0.194      -0.167       0.821
PC8           -0.5106      0.316     -1.616      0.107      -1.132       0.111
PC9           -1.2705      0.313     -4.060      0.000      -1.886      -0.655
PC10           0.6292      0.327      1.922      0.055      -0.015       1.273
PC11          -0.0189      0.270     -0.070      0.944      -0.550       0.512
==============================================================================
Omnibus:                       28.505   Durbin-Watson:                   2.110
Prob(Omnibus):                  0.000   Jarque-Bera (JB):               33.089
Skew:                          -0.734   Prob(JB):                     6.53e-08
Kurtosis:                       3.348   Cond. No.                         148.
==============================================================================

Notes:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.
MSE:
 92.68730265317721

QQ plot

The following figure shows the QQ plots of the four regression models described above. From the QQ plots of the four models, we can assess whether the residuals approximate a normal distribution:

• Original data (excluding MS, G1, and G2) + regression: best overall fit, but the right tail deviates significantly, indicating large prediction errors for extremely high scores.
• MDS + regression: a simpler model, but the tails of the residuals deviate more, suggesting the influence of outliers.
• MDS + PCA + regression: slight improvement in tail deviation, but the middle-to-tail range still does not align with the ideal line; residuals are still not ideal.
• MDS + PCA (80% explained variance) + regression: the most noticeable deviation from normality, especially in the right tail, likely due to excessive information compression in PCA.

None of the four models fully satisfy the assumption of normally distributed residuals, but the model using the original dataset comes closest.

Stepwise Variable Selection

Do you remember our goal? We aim to identify which variables influence G3. Therefore, we will gradually add variables into the regression model and evaluate whether each added variable significantly improves the model’s ability to fit G3.

Sound familiar? That’s right, we plan to implement a procedure similar to the step() function in R. The following section introduces an implementation of bidirectional elimination stepwise regression, which will be used to select influential variables.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

# Stepwise variables selection
import statsmodels.api as sm

def bidirectional_elimination(X, y, p_threshold=0.05):
    selected_feature = []
    remaining_feature = list(X.columns)
    best_aic = float('inf')
    
    while remaining_feature or selected_feature:
        improved = False
        
        # Forward Step
        forward_candidates = []
        for f in remaining_feature:
            model = sm.OLS(y, sm.add_constant(X[selected_feature + [f]])).fit()
            forward_candidates.append((model.aic, f))
        
        forward_candidates.sort()
        best_forward_aic, best_candidate = forward_candidates[0]
        
        if best_forward_aic < best_aic:
            best_aic = best_forward_aic
            selected_feature.append(best_candidate)
            remaining_feature.remove(best_candidate)
            improved = True
        
        # Backward Step
        model = sm.OLS(y, sm.add_constant(X[selected_feature])).fit()
        p_values = model.pvalues.iloc[1:]  # ignore intercept
        max_p_value = p_values.max()
        
        if max_p_value > p_threshold:
            worst_feature = p_values.idxmax()
            selected_feature.remove(worst_feature)
            remaining_feature.append(worst_feature)
            improved = True
        
        if not improved:
            break
            
    return selected_feature

Stepwise regression consists of three approaches:

• Forward selection: The regression model starts with no variables. Then, variables are added one by one based on their statistical significance, forming the final set of predictors.

• Backward elimination: All potential predictors are included at the beginning. Then, variables are removed one by one based on their lack of contribution to the model.

• Bidirectional elimination: Similar to forward selection, variables are added step by step. However, any variable that becomes insignificant during the process will also be removed.

By setting a threshold of p-value = 0.05 for variable selection, the resulting variables from each model are as follows:

PCA Again

Here, we perform PCA once more, this time using only the variables selected from the MDS + stepwise regression process. The results are as follows:

From the previous table, it’s not easy to clearly understand the relationships between variables. Therefore, we again performed PCA on the selected variables and plotted the explained variance. It can be observed that at least PC7 is needed to explain over 80% of the cumulative variance.

Next, we plotted the sample scatterplots of PC1 vs. PC2 using both the original dataset and the PCA after variable selection, with G3 scores used for color mapping. To better highlight the contrast between high and low scores:

• Scores from 15 to 20 are labeled as high and colored red.
• Scores from 0 to 5 are labeled as low and colored blue.

In the figure above:

• The left plot shows the result of PCA on the cleaned original dataset.
• The right plot shows PCA results after stepwise variable selection (SW PCA).

PCA: PC1 vs. PC2

• High-, mid-, and low-score samples are mixed and do not form clear clusters or boundaries.
• Although PC1 and PC2 capture the most total variance, they may not be directly related to G3.
• This suggests that PCA on all original variables may not provide strong discriminative power for G3.

SW PCA: PC1 vs. PC2

• High-score samples are mainly concentrated in the positive PC1 region.
• Low-score samples are mainly concentrated in the negative PC1 region.

This indicates that performing PCA after variable selection enables principal components, especially PC1, to better distinguish between different performance levels.

QQ Plot

After selecting variables, we conducted regression analysis and plotted the QQ plots for four models, as shown below:

Model Comparison

It is well known that when applying PCA, selecting only the top n% of components inevitably leads to information loss compared to using all principal components. Therefore, we exclude such models from further discussion here.

The following section compares the six models previously introduced.

From the table above, we observe that the model using the original dataset (excluding MS, G1, and G2) + regression demonstrates the best fit for predicting G3. It achieves the highest explanatory power, as indicated by both $R^2$ and adjusted $R^2$, and also yields the lowest mean squared error (MSE), suggesting that it captures the data variability well and offers more accurate predictions.

On the other hand, the original dataset (excluding MS, G1, and G2) + stepwise regression and MDS + PCA + stepwise regression models have relatively low P-values. This indicates that the variables selected in these models have statistically significant effects on G3, meaning the models are effective at identifying predictors with strong explanatory power.

When comparing AIC values, the original dataset + stepwise regression model has the lowest AIC, suggesting it offers the best overall performance by balancing explanatory strength and model complexity.

The following table shows the top three influential variables for each model.

In summary, we select the Original data (excluding MS, G1, and G2) + Stepwise Regression model as the model for evaluating G3. From this model, we find that the variables influencing students are Mjob and failures, indicating that family background and student performance in school courses are key factors affecting the final grades.

Conclusion

Through this collaborative data analysis, we have learned many analytical methods that we had not previously encountered. Through brainstorming, we were able to uncover hidden yet significant information within the data. Since the dataset mainly consists of categorical variables, a key challenge in this project was how to properly transform these into numerical variables for further analysis. Improper transformation may lead to information loss; therefore, selecting appropriate processing methods was crucial and represents an important data analysis skill we have acquired.

We would like to express our gratitude to Wang, Xuan-Chun and Sin, Wen-Lee for their assistance in the analysis, which allowed this project to proceed smoothly.

Further Learning

• The ipynb file used in this article.
• The Github repository for this article.

References

• Adil Shamim. (April 2025). Math-Students Performance Data. Kaggle. Retrieved on May 1, 2025, from https://www.kaggle.com/datasets/adilshamim8/math-students

• Cortez, P. (2008). Student Performance [Dataset]. UCI Machine Learning Repository. from https://doi.org/10.24432/C5TG7T

• Cortez, P., & Silva, A. M. (2008). Using data mining to predict secondary school student performance.