Definition of Dataset

The cover image is a dataset definition illustration generated by ChatGPT, using the prompt “A flat-style digital illustration representing the concept of “dataset definition”, without any text. The scene features a computer monitor displaying a simple dataset table (columns and rows), surrounded by minimalistic icons such as a database, document, network, and a checkmark. These elements are connected with dotted lines, symbolizing relationships and data structure. The color palette is clean and modern, using tones like teal, blue, and white. The overall composition follows a 16:9 aspect ratio, with a professional and conceptual data visualization feel.” .

Introduction

In this era of big data, our lives are filled with various forms of information. If we treat each piece of data as an observation and collect them together, we can roughly obtain a dataset composed of many individual data points. These datasets can be either structured or unstructured. By organizing and filtering the data to extract useful information and eliminate irrelevant parts, we can create a dataset suitable for analysis, model training, or academic research. A dataset may include different data types such as numbers, text, images, audio, or video, depending on its intended application and the methods used to collect and organize it.

Types of Datasets

Datasets can generally be categorized based on their structure and purpose as follows:

• Structured Dataset

  Data presented in forms such as tables or lists, where each record has fixed fields (features) and a consistent format, making it easy to store, query, and analyze.

  Examples: spreadsheets, database records, .csv files.

• Unstructured Dataset

  Data without a fixed structure and not directly representable as tables. This type of data typically requires specific preprocessing before it can be used.

  Examples: images, videos, text documents, novels.

• Labeled Dataset

  Each data point is associated with a label that helps models learn the relationship between input and output. Commonly used in supervised learning.

  Examples: handwritten digit recognition datasets, the Iris dataset.

• Unlabeled Dataset

  Data without accompanying labels, often requiring analysis such as clustering or dimensionality reduction. Commonly used in unsupervised learning.

  Examples: customer purchase records, climate data, water temperature data.

Dataset Splitting

When fitting a dataset using machine learning, we often split the original dataset to evaluate how well a model generalizes to new data. Since it may not be possible to obtain entirely new data in real-time for testing, we typically reserve a portion of the dataset in advance for testing purposes.

The simplest approach is to divide the dataset into two parts: a training set and a test set. However, this method does not allow the model to adjust based on testing results. Therefore, we often further split the training set to create a validation set for tuning model hyperparameters.

Training Set

The training set is the foundation for model learning. It allows the model to fit the data and identify the relationships between inputs and outputs.

A simple analogy: the training set is like a student learning in school through repeated practice.

Validation Set

The validation set is used to evaluate the model’s performance at the end of each training cycle and to tune hyperparameters.

It is like a student’s regular quizzes at school, helping teachers adjust teaching methods based on the student’s performance.

If no hyperparameter tuning is needed, the validation set may be omitted.

Test Set

The test set is used only after the model has been fully trained. Its purpose is to assess the model’s ability to generalize to completely unseen data. Unlike the validation set, the test set is not used for tuning.

In an analogy, the test set is like a final exam that evaluates the student’s overall learning for the semester.

Split Ratios

There is no strict rule for dataset splitting ratios. Common proportions include

Conclusion

Datasets are a crucial part of data science. Whether for model training, feature extraction, or research analysis, datasets are deeply intertwined with these processes. Learning about the types and splitting methods of datasets allows us to manage and process data more effectively, enhancing both model accuracy and practical utility.