Methods of Cross-Validation

The cover image was generated by ChatGPT using an image of cross-validation, with the following prompt: “A 2D vector graphic visually representing cross-validation in machine learning, with a light blue background. Include a clear title “CROSS-VALIDATION” at the top. Show a dataset divided into five folds labeled “Fold 1” to “Fold 5”. Use arrows to indicate training and validation data splits, and include labeled boxes for “Training Data”, “Validation Data”, and “Model”. Use clean lines, modern flat design, and contrasting colors for clarity. Aspect ratio 16:9.” 。

Introduction

Cross-validation (CV) is a commonly used model validation technique in machine learning, employed to assess a model’s generalization ability on unseen data. By splitting the dataset into training, validation, and test sets, and repeatedly conducting training and evaluation, cross-validation can effectively test a model’s performance even when the data is limited. It helps evaluate generalization ability and is useful in reducing issues like overfitting and bias.

Cross-validation plays a crucial role in controlling overfitting, selecting the best model, and tuning hyperparameters. In particular, it helps predict how a model will perform on new data, making it an indispensable part of modern machine learning workflows.

Model Generalization

Model generalization refers to a model’s ability to maintain good predictive performance when faced with new, previously unseen data. In other words, a model with good generalization not only performs well on the training data but can also be effectively applied to completely new data in real-world scenarios.

When training a model, if the model merely memorizes the details of the training data and fails to apply to new data, it leads to overfitting. Such models may perform excellently during the training phase but often perform poorly in practical applications.

In contrast, a model with good generalization ability can apply the knowledge it has learned to new data while maintaining stable and accurate predictions. This kind of model does not just memorize the characteristics of the training data but successfully captures the underlying patterns and relationships within the data. Models with strong generalization are more practical in real-world applications, as they can continue making reliable decisions or predictions regardless of changes in data sources.

Related Terms

• Overfitting: When a model overlearns the training data, losing its ability to make predictions on new data.

• Underfitting: When a model fails to learn adequately, resulting in poor performance even on the training data.

Dataset Splitting

When training a model, we typically split the dataset into a training set and a test set. During the training phase, the model is fitted only on the training set, while the test set remains unseen by the model, making it suitable for the final evaluation of the model’s performance.

However, during training, we often need to optimize certain parameters. In this case, we can further split a portion of the training set to create a validation set, which is used to tune the model so that its performance on the validation set is optimal.

To avoid overfitting the model to the validation set, we can use cross-validation to partition the training data into multiple groups. In each iteration, a different subset is used for training and another for evaluation. Repeating this process helps improve the model’s generalization ability.

Common Cross-Validation Methods

Hold-out Method

The Hold-out Method, also known as simple cross-validation, is the simplest validation method. However, since this method does not involve any crossing of data subsets, it is technically not classified as a type of cross-validation.

Method

In this approach, a certain proportion of the dataset is randomly selected to form the training set, while the remaining data is used as the test set. Common ratios include using 70% of the data for training and 30% for testing, or 90% for training and 10% for testing. However, this method has some drawbacks. For example:

Therefore, when using the hold-out method for model validation, it is crucial to ensure that the training set is sufficiently large and diverse to avoid distorted evaluation results. In general, the test set should not exceed one-third of the original dataset.

Once the training and test sets are separated, the model is trained using the training set, predictions are made using the test set, and the model is evaluated using appropriate performance metrics.

Advantages

• Simple and computationally inexpensive, making it suitable for cases involving large datasets where a quick estimate is sufficient.

Disadvantages

• The results heavily depend on a single random split, leading to unstable and potentially biased evaluation metrics. When the dataset is small, representativeness issues may arise.

Suitable Scenarios

• Quick assessments with large datasets, or situations where repeated evaluations can be averaged for more stable results.

K-fold Cross-Validation

K-fold cross-validation (KCV) improves upon the hold-out method by reducing the model’s dependency on a single random split between the training and test sets, providing a more stable evaluation of the model.

Method

First, K-fold cross-validation randomly splits the entire dataset into K equal-sized subsets (called folds). Each fold contains a portion of the dataset, ensuring that every sample has a chance to appear in both the training and test sets. In each validation round, one of the folds is used as the test set, while the remaining K−1 folds are combined to form the training set. This process is repeated K times, with a different fold used as the test set each time.

After K rounds of validation, we obtain K evaluation results. These are averaged (and the standard deviation is often calculated as well) to provide a more accurate assessment of the model’s generalization performance.

Advantages

• Each sample appears in both the training and test sets at least once, reducing the model’s reliance on a specific data split and minimizing bias and overfitting risks.
• Provides more stable results, avoiding the bias that can come from a single test split.
• Maximizes data usage efficiency, making it well-suited for small datasets.
• Common values for K are $K = 5$ or $K = 10$, which help achieve a better balance between bias and variance.

Disadvantages

• Requires training the model K times, which can be computationally expensive for large datasets.
• If the data split is not sufficiently random, bias may still occur.
• A very large K increases computational cost and may lead to overly similar training subsets, potentially increasing variance.

Suitable Scenarios

• Small datasets.
• Situations where objective evaluation of a model’s generalization ability is needed.

Stratified K-fold Cross-Validation

Stratified K-fold cross-validation (SKCV) is an improved version of K-fold cross-validation. Because K-fold cross-validation splits the dataset randomly, it may result in the following situation:

Since the data splitting process separates the classes, individual models during training may not learn the features of the classes that appear in the test set. This could lead to biased model training.

Method

First, the class distribution of the original dataset is determined. When dividing the data into subsets, the algorithm ensures that each fold maintains approximately the same proportion of classes as the original dataset.

After that, the process follows the same steps as standard K-fold cross-validation. In each iteration, one fold is used as the test set, while the remaining K−1 folds are combined as the training set. This process is repeated K times, with a different fold used as the test set each time.

After completing K iterations, the evaluation metrics are averaged (and standard deviation may be calculated) to more accurately assess the model’s generalization performance.

Advantages

• Prevents class imbalance in the data splits.
• Better reflects the model’s performance across different classes, especially when the class distribution is highly imbalanced.
• Improves the stability and credibility of evaluation results.

Disadvantages

• Only applicable to supervised learning tasks where labels or class attributes are available.
• When the dataset is very small or some classes have too few samples, some folds may not contain all classes, leading to inaccurate evaluation.
• Slightly more complex to implement and more time-consuming than simple random splitting, which can be a burden for large datasets.

Suitable Scenarios

• Binary or multi-class classification problems with imbalanced class distributions, such as medical diagnosis (disease vs. no disease), fraud detection, or minority class detection.
• Cases where model evaluation requires more stable and fair comparisons.
• Datasets where certain classes have a very small representation.

Leave-One-Out Cross-Validation

Leave-One-Out Cross-Validation (LOOCV) is a special form of K-fold cross-validation where each iteration uses only one sample from the dataset as the test set, while all the remaining data is used as the training set. If the dataset contains $n$ samples, LOOCV will perform $n$ rounds of training and validation. Since every single sample serves as the test set once, LOOCV makes full use of the available data and provides a highly detailed evaluation of the model’s generalization ability.

Method

Assume the dataset contains $n$ samples. In each iteration, one sample is selected as the test set, and the remaining $n - 1$ samples are used to train the model. This process is repeated $n$ times, each time with a different sample used as the test set.

After $n$ iterations, the evaluation metrics from each run are averaged (and standard deviation is often calculated), resulting in a more stable and lower-bias estimate compared to standard K-fold cross-validation.

Advantages

• Maximizes utilization of the dataset.
• No need for random splitting, and the results are not influenced by how the data is divided.
• Typically yields lower estimation bias compared to K-fold cross-validation.

Disadvantages

• Requires retraining the model for each sample, leading to high computational cost, unsuitable for large datasets.

Suitable Scenarios

• Small datasets.
• Scenarios requiring precise model evaluation, such as academic research or experimental settings.

Leave-P-Out Cross-Validation

Leave-P-Out Cross-Validation (LPOCV) is a generalization of Leave-One-Out Cross-Validation (LOOCV). Unlike LOOCV, which selects only one data point as the test set each time, LPOCV selects $p$ samples from the dataset as the test set and uses the remaining $n - p$ samples for training. When $p = 1$, LPOCV becomes equivalent to LOOCV.

Method

Assume the dataset contains $n$ samples. In each iteration, $p$ samples are selected as the test set, and the remaining $n - p$ samples form the training set. This process is repeated across multiple combinations of test sets.

After training the model on all selected combinations, the evaluation metrics are averaged and their standard deviation is calculated to assess the model’s generalization ability.

Theoretically, LPOCV would involve $\frac{n!}{(n - p)! p!}$ combinations. While this equals $n$ when $p = 1$, it grows rapidly as $p > 1$, making it computationally infeasible. In practice, random repeated sampling is often used instead, ensuring each data point has multiple chances to appear in the test set while keeping training costs manageable.

Advantages

• Flexibility to choose $p$ based on dataset size and needs, balancing training and test set sizes.
• Evaluates model generalization more precisely by testing across numerous data subsets.
• Approximates an almost unbiased estimate by covering all possible sample combinations.

Disadvantages

• For $p > 1$ and large datasets, the number of combinations grows explosively, resulting in excessive training cost.
• In practice, not all combinations can be evaluated. Typically only a subset is randomly sampled, which may affect stability and reproducibility.
• Reusing the same data points across many tests can lead to correlated validation errors, complicating statistical inference.

Suitable Scenarios

• Small datasets.
• Situations requiring insight into the model’s stability across different data combinations.
• Domains requiring high-precision analysis with limited data, such as medical research, biostatistics, or psychology.

Repeated K-fold Cross-Validation

Repeated K-fold Cross-Validation (RKCV) is a variant of K-fold cross-validation that aims to improve the stability and accuracy of model evaluation by repeating the K-fold process multiple times. This helps reduce the variance caused by random splits of the dataset.

Method

As with standard K-fold cross-validation, the dataset is randomly divided into K subsets. One subset is used as the test set, and the remaining K − 1 subsets are combined as the training set. This process is repeated K times, with each subset used once as the test set.

After completing one round of K-fold cross-validation, the dataset is randomly reshuffled and split into K new subsets, and the process is repeated. This full K-fold validation is typically repeated 5 to 10 times, especially for small datasets, to improve the robustness of the evaluation.

In the end, the performance metrics from all repetitions are averaged, and their standard deviations are computed to estimate the model’s generalization ability.

Advantages

• Repeating the validation multiple times reduces variance caused by a single data split and improves result stability.
• By testing the model across various randomly selected subsets, it better reflects the model’s generalization capacity.
• Helps reduce the risk of overfitting.

Disadvantages

• Repeating the K-fold process multiple times significantly increases the number of training sessions and the overall computational cost.
• Each repetition involves K separate train-test cycles, leading to longer runtime.
• Different splits in each repetition may require more detailed analysis of the results.

Suitable Scenarios

• Small datasets.
• Datasets with high variability in distribution.

Group K-fold Cross-Validation

When the dataset contains inherent group dependencies, such as records from the same user, patient, device, or time period, applying standard K-fold cross-validation directly to individual samples may cause data from the same group to appear in both the training and test sets. This leads to data leakage, resulting in overly optimistic model performance estimates.

To address this, Group K-fold Cross-Validation (GKCV) ensures that the same group does not appear in both the training and test sets. This reduces the risk of overfitting, avoids bias caused by clustered test data, and improves the model’s ability to generalize to unseen groups.

For example, if there are three categories of data, the test set will sample from three distinct groups in each fold, and these groups will be mutually exclusive across all folds.

Method

First, specify the group to which each data point belongs, such as user ID, device number, or the same school.

Next, split the dataset based on these groupings and divide the data into multiple distinct groups.

Then, divide the dataset into K subsets, where each subset consists of multiple groups, ensuring that the groups do not overlap between subsets.

Select one of the subsets as the test set, and combine the remaining K − 1 subsets as the training set. Repeat this process K times, selecting a different subset as the test set each time.

After K rounds of testing, calculate the average and standard deviation of the evaluation metrics to more accurately assess the model’s generalization performance.

Advantages

• Effectively prevents data leakage between groups.
• Enhances model generalization to unseen groups.
• Test sets better reflect the dataset’s diversity and distribution.

Disadvantages

• Insufficient number of groups can limit the effectiveness of cross-validation.
• Uneven group sizes may cause imbalanced folds.
• Slightly more complex than standard K-fold validation. Requires clearly defined group labels for each sample.

Suitable Scenarios

• Any situation involving group-based data where data leakage is a concern.

Nested Cross Validation

Nested Cross-Validation (Nested CV) is a variant of K-fold cross validation. It was developed to avoid tuning hyperparameters and evaluating model performance on the same test set, which can lead to overly optimistic results.

By introducing a two-layer cross validation structure, Nested Cross Validation separates hyperparameter selection from model evaluation, effectively preventing data leakage and providing a more reliable assessment of the model.

Method

Nested Cross-Validation consists of an outer loop and an inner loop. The outer loop is responsible for evaluating the model, while the inner loop is used for selecting the best hyperparameters.

To perform Nested CV, start by defining the outer loop. The outer loop works the same way as K-fold cross validation: split the dataset into K subsets, select one subset as the test set, and use the remaining K − 1 subsets as the training set.

Next, define the inner loop. For each training set in the outer loop, apply a cross-validation method again, such as another K-fold cross-validation, splitting the data into a new training set and validation set. Use these to iteratively evaluate different combinations of hyperparameters.

In each round of the outer loop, identify the hyperparameters that performed best in the inner loop. Then, retrain the model using these hyperparameters on the entire outer training set, and evaluate it on the outer test set.

Repeat this process until all K subsets have served as the outer test set. Finally, compute the average and standard deviation of the evaluation metrics to estimate the model’s generalization ability.

Advantages

• Prevents data leakage that could distort model evaluation.
• Enhances the model’s ability to generalize to unseen data.
• Ensures hyperparameter selection is independent of the test data, enabling more objective tuning.

Disadvantages

• The dual-layer structure requires many rounds of training, making it computationally expensive.
• Requires clear separation of logic and data between the inner and outer loops.
• Not suitable for situations requiring real-time model evaluation.
• Not suitable for large datasets.

Suitable Scenarios

• When precise hyperparameter tuning is required.
• When the dataset is prone to overfitting or data leakage.

Monte Carlo Cross-Validation

Monte Carlo Cross-Validation (MCCV), also known as Randomized Cross-Validation (RCV), is a method for evaluating a model’s generalization ability by repeatedly and randomly splitting the dataset. Unlike K-fold cross-validation, MCCV does not guarantee that every data point will be used, nor that the training or test set sizes will be exactly the same in each round. A single data point may appear in the test set multiple times, or never at all.

MCCV can be seen as a repeated application of the holdout method. Because the splitting strategy is highly flexible, it also introduces larger evaluation variance.

Method

First, define the ratio between the training set and test set, as well as the number of repetitions.

Next, repeatedly perform random splits of the dataset into a training set and a test set, then train and evaluate the model.

Once completed, compute the average and standard deviation of the evaluation metrics to assess the model’s performance.

Advantages

• Flexible adjustment of training/test set ratios and number of repetitions.
• No need to create fixed K-fold partitions or handle group constraints.

Disadvantages

• Some data points may never be used for testing, leading to biased evaluation.
• Results may vary due to randomness in each split, requiring multiple runs for stability.
• The same data point may appear in multiple test sets, making them not mutually independent.

Suitable Scenarios

• When the dataset is large enough to allow for repeated random sampling.
• When a quick estimate of model performance is desired without complex grouping strategies.
• Large-scale datasets.

Time Series Cross-Validation

Time series data has a clear chronological order, and future data should never be used to predict the past. If standard methods like K-fold cross-validation are applied to time series data, it may disrupt the temporal sequence, leading to data leakage and unreliable evaluation results.

Time Series Cross-Validation (TSCV) is a validation strategy that preserves the chronological order and ensures that only past data is used to predict the future, effectively preventing future information from leaking into the training set.

Method

Two commonly used strategies are expanding window and sliding window.

Expanding Window

First, sort the data by time and define the initial sizes of the training and test sets. The test set size remains fixed. The training set includes all data up to a certain time point, and the test set contains the data immediately following that point.

Then, incrementally expand the training window forward in time, train the model, and shift the test set accordingly. Repeat this process until all available data has been used.

Sliding Window

First, sort the data by time and define fixed sizes for both the training and test sets. The training set includes data up to a certain time point, and the test set contains the data immediately after that point.

Then, slide both the training and test windows forward in time and repeat the training and evaluation steps until all data has been used.

Advantages

• Preserves temporal order and prevents future information leakage.
• Closely resembles real-world prediction and deployment scenarios.
• Flexible adjustment of training/test set ratios and window sizes.

Disadvantages

• Cannot randomly shuffle data.
• If data volume is small, test samples may be insufficient.
• For long time series, computational cost can be high.

Suitable Scenarios

• Time series analysis, such as in stock markets, weather forecasting, or sales data.
• When models must strictly follow a “train-on-past, predict-the-future” rule.
• When forecasting future trends, events, or behaviors.

Method Comparison

Conclusion

Cross-validation is a core method for evaluating a model’s generalization ability. With varying dataset sizes and characteristics, we should select the appropriate cross-validation technique based on the actual situation to avoid overestimating or underestimating model performance. As models become increasingly complex and data volumes continue to grow, cross-validation remains a foundational practice in modern machine learning—an essential tool for building robust and generalizable models.

References

• 10程式中. (October 8, 2021). [Day 26] 交叉驗證 K-Fold Cross-Validation. iT邦幫忙. Retrieved May 1, 2025, from https://ithelp.ithome.com.tw/articles/10279240

• 小千北同学超爱写代码. (January 19, 2021). StratifiedKFold和KFold的区别（几种常见的交叉验证）. 博客园. Retrieved May 1, 2025, from https://www.cnblogs.com/liuxiangyan/p/14299865.html

• 什么是：留P交叉验证. (n.d.). 轻松学习统计. Retrieved May 1, 2025, from https://zh-cn.statisticseasily.com/glossario/what-is-leave-p-out-cross-validation/

• 西红柿炒豆腐. (October 19, 2020). K-fold vs. Monte Carlo cross-validation（K折交叉验证与蒙特卡洛交叉验证（MCCV））. CSDN. Retrieved May 1, 2025, from https://blog.csdn.net/weixin_44839513/article/details/109163111

• Cross-validation (statistics). (February 19, 2025). Wikipedia, the free encyclopedia. Retrieved May 1, 2025, from https://en.wikipedia.org/wiki/Cross-validation_(statistics)

• 李航. (2022). 機器學習聖經：最完整的統計學習方法（初版）. 深智數位股份有限公司. Accessed May 1, 2025.

• 新语数据故事汇. (December 9, 2023). 一文带您了解交叉验证(Cross-Validation):数据科学家必须掌握的7种交叉验证技术. 知乎. Retrieved May 1, 2025, from https://zhuanlan.zhihu.com/p/671321268

• 数学人生. (November 7, 2015). 留一法交叉验证和普通交叉验证有什么区别？ 知乎. Retrieved May 1, 2025, from https://www.zhihu.com/question/23561944

• 劉智皓. (February 7, 2021). 機器學習_學習筆記系列(13)：交叉驗證(Cross-Validation)和MSE、MAE、R2. Medium. Retrieved May 1, 2025, from https://tomohiroliu22.medium.com/機器學習-學習筆記系列-13-交叉驗證-cross-validation-和mse-mae-r2-bc8fef393f7c

• Claire Chang. (June 21, 2023). k-Fold Cross-Validation（交叉驗證）. Claire’s Blog. Retrieved May 1, 2025, from https://claire-chang.com/2023/06/21/k-fold-cross-validation/

• Cynthia. (February 20, 2025). 訓練集、驗證集、測試集的定義與劃分. HackMD. Retrieved May 1, 2025, from https://hackmd.io/@CynthiaChuang/What-is-the-Difference-between-Training-Validation-and-Test-Dataset

• jingsongs. (September 8, 2017). KFold，StratifiedKFold k折交叉切分. CSDN. Retrieved May 1, 2025, from https://blog.csdn.net/wqh_jingsong/article/details/77896449

• ljalphabeta. (n.d.). K折交叉验证评估模型性能. Gitbook. Retrieved May 1, 2025, from https://ljalphabeta.gitbooks.io/python-/content/kfold.html

• Hastie, T., Tibshirani, R., & Friedman, J. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction (2nd ed.). Springer. Retrieved May 1, 2025, from https://hastie.su.domains/ElemStatLearn/

• James, G., Witten, D., Hastie, T., & Tibshirani, R. (2021). An Introduction to Statistical Learning: With Applications in R (2nd ed.). Springer. Retrieved May 1, 2025, from https://www.statlearning.com/

• Kohavi, R. (1995). A study of cross-validation and bootstrap for accuracy estimation and model selection. In Proceedings of the 14th International Joint Conference on Artificial Intelligence (IJCAI) (Vol. 2, pp. 1137–1143). Retrieved May 1, 2025, from https://www.ijcai.org/Proceedings/95-2/Papers/016.pdf