1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
| Intializing conda
Activating Conda Env: sssd
[Execution - Training]
/content/drive/MyDrive/Colab Notebooks/SSSD_CP/scripts/diffusion/train.py --model_config configs/model.yaml --training_config configs/training.yaml
2025-03-21 07:37:45,643 - sssd.utils.logger - INFO - Model spec: {'wavenet': {'input_channels': 14, 'output_channels': 14, 'residual_layers': 36, 'residual_channels': 256, 'skip_channels': 256, 'diffusion_step_embed_dim_input': 128, 'diffusion_step_embed_dim_hidden': 512, 'diffusion_step_embed_dim_output': 512, 's4_max_sequence_length': 100, 's4_state_dim': 64, 's4_dropout': 0.0, 's4_bidirectional': True, 's4_use_layer_norm': True}, 'diffusion': {'T': 200, 'beta_0': 0.0001, 'beta_T': 0.02}}
2025-03-21 07:37:45,644 - sssd.utils.logger - INFO - Training spec: {'batch_size': 80, 'output_directory': './results/checkpoint', 'ckpt_iter': 'max', 'iters_per_ckpt': 10, 'iters_per_logging': 10, 'n_iters': 60000, 'learning_rate': 0.0002, 'only_generate_missing': True, 'use_model': 2, 'masking': 'forecast', 'missing_k': 24, 'data': {'train_path': './datasets/Mujoco/train_mujoco.npy'}}
2025-03-21 07:37:45,660 - sssd.utils.logger - INFO - Using 1 GPUs!
2025-03-21 07:37:45,679 - sssd.utils.logger - INFO - Output directory ./results/checkpoint/T200_beta00.0001_betaT0.02
2025-03-21 07:38:04,809 - sssd.utils.logger - INFO - Current time: 2025-03-21 07:38:04
2025-03-21 07:38:05,557 - sssd.utils.logger - INFO - No valid checkpoint model found, start training from initialization.
2025-03-21 07:38:05,557 - sssd.utils.logger - INFO - Start the 1 iteration
100% 100/100 [03:33<00:00, 2.13s/it]
100% 100/100 [03:33<00:00, 2.14s/it]
100% 100/100 [03:33<00:00, 2.13s/it]
100% 100/100 [03:33<00:00, 2.13s/it]
100% 100/100 [03:33<00:00, 2.13s/it]
100% 100/100 [03:33<00:00, 2.13s/it]
100% 100/100 [03:33<00:00, 2.13s/it]
100% 100/100 [03:33<00:00, 2.14s/it]
100% 100/100 [03:33<00:00, 2.13s/it]
100% 100/100 [03:33<00:00, 2.13s/it]
2025-03-21 08:13:41,344 - sssd.utils.logger - INFO - Iteration: 10 Loss: 0.024617835879325867
100% 100/100 [03:33<00:00, 2.14s/it]
100% 100/100 [03:33<00:00, 2.13s/it]
100% 100/100 [03:33<00:00, 2.13s/it]
100% 100/100 [03:33<00:00, 2.14s/it]
100% 100/100 [03:33<00:00, 2.14s/it]
100% 100/100 [03:33<00:00, 2.13s/it]
100% 100/100 [03:33<00:00, 2.13s/it]
100% 100/100 [03:33<00:00, 2.13s/it]
100% 100/100 [03:33<00:00, 2.14s/it]
100% 100/100 [03:33<00:00, 2.13s/it]
2025-03-21 08:49:23,932 - sssd.utils.logger - INFO - Iteration: 20 Loss: 0.023843389004468918
100% 100/100 [03:33<00:00, 2.14s/it]
100% 100/100 [03:33<00:00, 2.13s/it]
100% 100/100 [03:33<00:00, 2.14s/it]
100% 100/100 [03:33<00:00, 2.13s/it]
100% 100/100 [03:33<00:00, 2.13s/it]
100% 100/100 [03:33<00:00, 2.13s/it]
100% 100/100 [03:33<00:00, 2.14s/it]
100% 100/100 [03:33<00:00, 2.14s/it]
100% 100/100 [03:33<00:00, 2.14s/it]
100% 100/100 [03:33<00:00, 2.13s/it]
2025-03-21 09:25:07,505 - sssd.utils.logger - INFO - Iteration: 30 Loss: 0.01294409018009901
100% 100/100 [03:33<00:00, 2.14s/it]
100% 100/100 [03:33<00:00, 2.14s/it]
100% 100/100 [03:33<00:00, 2.14s/it]
100% 100/100 [03:33<00:00, 2.14s/it]
100% 100/100 [03:33<00:00, 2.13s/it]
100% 100/100 [03:33<00:00, 2.13s/it]
100% 100/100 [03:33<00:00, 2.13s/it]
100% 100/100 [03:33<00:00, 2.14s/it]
100% 100/100 [03:33<00:00, 2.14s/it]
100% 100/100 [03:33<00:00, 2.13s/it]
2025-03-21 10:00:49,689 - sssd.utils.logger - INFO - Iteration: 40 Loss: 0.009161572903394699
100% 100/100 [03:33<00:00, 2.14s/it]
100% 100/100 [03:33<00:00, 2.13s/it]
30% 30/100 [01:03<02:29, 2.13s/it]
|
