20250318 meeting

SSSD_CP repository exploration

Published 2025-03-12 Updated 2025-03-18 1016 words 5 minutes views

Contents

Introduction

While reading and testing the SSSD_CP repository, since the SSSD branch requires Linux, Conda, and Docker, I began with the installation. Since the operating system is Windows, I chose to install Windows Subsystem for Linux (WSL) as the operating environment. During the testing process, I discovered that Windows uses the line ending \r\n, commonly known as CRLF, whereas Linux, Unix, and macOS systems use \n as the line ending, known as LF. Therefore, when a file edited on Windows is executed in WSL, it fails to run correctly (because the extra \r causes file path errors).

As a result, we need to use the dos2unix Linux utility to convert the line endings in the files to the characters accepted by Linux before running the .sh scripts.

Training Execution

Below is the configuration file for training using the mujoco.npy dataset.

model.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
wavenet:
  # WaveNet model parameters
  input_channels: 14  # Number of input channels
  output_channels: 14  # Number of output channels
  residual_layers: 36  # Number of residual layers
  residual_channels: 256  # Number of channels in residual blocks
  skip_channels: 256  # Number of channels in skip connections

  # Diffusion step embedding dimensions
  diffusion_step_embed_dim_input: 128  # Input dimension
  diffusion_step_embed_dim_hidden: 512  # Middle dimension
  diffusion_step_embed_dim_output: 512  # Output dimension

  # Structured State Spaces sequence model (S4) configurations
  s4_max_sequence_length: 100  # Maximum sequence length
  s4_state_dim: 64  # State dimension
  s4_dropout: 0.0  # Dropout rate
  s4_bidirectional: true  # Whether to use bidirectional layers
  s4_use_layer_norm: true  # Whether to use layer normalization

diffusion:
  # Diffusion model parameters
  T: 200  # Number of diffusion steps
  beta_0: 0.0001  # Initial beta value
  beta_T: 0.02  # Final beta value

In this, I only changed the input_channels and output_channels to match the number of features in the dataset for input and output. I also modified the s4_max_sequence_length to match the time points.

training.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
# Training configuration
batch_size: 80  # Batch size
output_directory: "/mnt/d/Code/sssd_cp_learning_and_testing/learning_and_testing/results/checkpoint"  # Output directory for checkpoints and logs
ckpt_iter: "max"  # Checkpoint mode (max or min)
iters_per_ckpt: 1000  # Checkpoint frequency (number of epochs)
iters_per_logging: 1000  # Log frequency (number of iterations)
n_iters: 60000  # Maximum number of iterations
learning_rate: 0.0002  # Learning rate

# Additional training settings
only_generate_missing: true  # Generate missing values only
use_model: 2  # Model to use for training
masking: "forecast"  # Masking strategy for missing values
missing_k: 24  # Number of missing values

# Data paths
data:
  train_path: "/mnt/d/Code/sssd_cp_learning_and_testing/learning_and_testing/SSSD/datasets/Mujoco/train_mujoco.npy"  # Path to training data

Below are the prompts for successfully executing the training script.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
user@LAPTOP-KOPTLCHM:/mnt/d/Code/sssd_cp_learning_and_testing/learning_and_testing/SSSD_CP$ conda activate sssd
(sssd) user@LAPTOP-KOPTLCHM:/mnt/d/Code/sssd_cp_learning_and_testing/learning_and_testing/SSSD_CP$ ./scripts/diffusion/training_job.sh -m configs/model.yaml -t configs/training.yaml
Script is running from: /mnt/d/Code/sssd_cp_learning_and_testing/learning_and_testing/SSSD_CP/scripts/diffusion
Intializing conda
Activating Conda Env: sssd
[Execution - Training]
/mnt/d/Code/sssd_cp_learning_and_testing/learning_and_testing/SSSD_CP/scripts/diffusion/train.py --model_config configs/model.yaml --training_config configs/training.yaml
2025-03-16 16:51:31,034 - sssd.utils.logger - INFO - Model spec: {'wavenet': {'input_channels': 14, 'output_channels': 14, 'residual_layers': 36, 'residual_channels': 256, 'skip_channels': 256, 'diffusion_step_embed_dim_input': 128, 'diffusion_step_embed_dim_hidden': 512, 'diffusion_step_embed_dim_output': 512, 's4_max_sequence_length': 100, 's4_state_dim': 64, 's4_dropout': 0.0, 's4_bidirectional': True, 's4_use_layer_norm': True}, 'diffusion': {'T': 200, 'beta_0': 0.0001, 'beta_T': 0.02}}
2025-03-16 16:51:31,034 - sssd.utils.logger - INFO - Training spec: {'batch_size': 80, 'output_directory': '/mnt/d/Code/sssd_cp_learning_and_testing/learning_and_testing/results/checkpoint', 'ckpt_iter': 'max', 'iters_per_ckpt': 1000, 'iters_per_logging': 1000, 'n_iters': 60000, 'learning_rate': 0.0002, 'only_generate_missing': True, 'use_model': 2, 'masking': 'forecast', 'missing_k': 24, 'data': {'train_path': '/mnt/d/Code/sssd_cp_learning_and_testing/learning_and_testing/SSSD/datasets/Mujoco/train_mujoco.npy'}}
2025-03-16 16:51:31,190 - sssd.utils.logger - INFO - Using 1 GPUs!
2025-03-16 16:51:31,287 - sssd.utils.logger - INFO - Output directory /mnt/d/Code/sssd_cp_learning_and_testing/learning_and_testing/results/checkpoint/T200_beta00.0001_betaT0.02
2025-03-16 16:51:42,974 - sssd.utils.logger - INFO - Current time: 2025-03-16 16:51:42
2025-03-16 16:51:44,226 - sssd.utils.logger - INFO - No valid checkpoint model found, start training from initialization.
2025-03-16 16:51:44,227 - sssd.utils.logger - INFO - Start the 1 iteration
  3%|███▌                                                                                                                   | 3/100 [01:46<56:43, 35.09s/it

It is important to note that this project must use Python 3.10.13 (to be compatible with the version of PyTorch in the project). Miniconda and Docker must also be installed beforehand, and the /envs/conda/build_conda_env.sh and /envs/vm/install_on_ubuntu.sh scripts should be executed to set up the environment.

Please remember to activate the conda sssd environment before running the scripts, or it will result in an error.

1
2
3
4
5
(base) user@LAPTOP-KOPTLCHM:/mnt/d/Code/sssd_cp_learning_and_testing/learning_and_testing/SSSD_CP$ conda deactivate
user@LAPTOP-KOPTLCHM:/mnt/d/Code/sssd_cp_learning_and_testing/learning_and_testing/SSSD_CP$ ./scripts/diffusion/training_job.sh -m configs/model.yaml -t configs/training.yaml
Script is running from: /mnt/d/Code/sssd_cp_learning_and_testing/learning_and_testing/SSSD_CP/scripts/diffusion
/mnt/d/Code/sssd_cp_learning_and_testing/learning_and_testing/SSSD_CP/scripts/diffusion/../../envs/conda/utils.sh: line 12: conda: command not found
No Conda environment found matching

Tip

If the execution is unsuccessful or if you receive a prompt indicating that the sssd module cannot be found, it is recommended to re-enter WSL and re-enter the conda sssd environment.

Observation

The missing_k here refers to

1
missing_k (int): The number of the last elements to be predicted.

After examining /sssd/core/utils.py, we can see that it includes additional handling for missing values, as shown below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
def get_mask_forecast(sample: torch.Tensor, k: int) -> torch.Tensor:
    """
    Get mask of same segments (black-out missing) across channels based on k.

    Args:
        sample (torch.Tensor): Tensor of shape [# of samples, # of channels].
        k (int): Number of missing values.

    Returns:
        torch.Tensor: Mask of sample's shape where 0's indicate missing values to be imputed, and 1's indicate preserved values.
    """
    mask = torch.ones_like(sample)  # Initialize mask with all ones

    # Calculate the indices of missing values
    s_nan = torch.arange(mask.shape[0] - k, mask.shape[0])

    # Apply mask for each channel
    for channel in range(mask.shape[1]):
        mask[s_nan, channel] = 0

    return mask

Training Results

Since a single iteration takes about an hour to train (on my laptop), here is the following:

https://raw.githubusercontent.com/Josh-test-lab/website-assets-repository/refs/heads/main/posts/1140318%20meeting/To%20be%20continued.jpg — To be continued.

Environment

Operating System: Windows 11 24H2
Subsystem: Windows Subsystem for Linux (Ubuntu)
Miniconda
Docker
CUDA 12.8 driver (Running version 12.1 in code)
Programming Language: Python 3.10.13 for Linux

Further Learning

I tested this project on the GitHub repository.

References

Juan Lopez Alcaraz, Nils Strodthoff. (2022). Diffusion-based time series imputation and forecasting with structured state space models. Transactions on Machine Learning Research. Referenced from https://openreview.net/forum?id=hHiIbk7ApW
SSSD. (2022). GitHub. Referenced from https://github.com/AI4HealthUOL/SSSD
SSSD_CP. (2024). GitHub. Referenced from https://github.com/egpivo/SSSD_CP

Contents

20250318 meeting

SSSD_CP repository exploration

Introduction

Training Execution

Observation

Training Results

Environment

Further Learning

References

Related Posts