Articles
Posts Tags Categories Random
Partners
About
About Author Portfolio Terms of Use Privacy Policy Time Travel Donate
More
Settings Funny Tools Useful Websites Student Works Message Board Recent Comments

Contents

20250422 meeting

 Included in Meeting
 Published   Updated    6420 words   31 minutes   views

Introduction

The initial plan for this experiment was to use real-time air quality data from the Ministry of Environment for analysis and data imputation. However, due to the excessive amount of missing data, the dataset was difficult to clean and thus unsuitable for testing the feasibility of the code. Therefore, we later switched to using air quality data from Sanyi Township, Miaoli County.

Data Processing

The following is the original code used to process the Ministry of Environment data. All data will be converted into .npy files.

Real-Time Data from the Ministry of Environment

Define functions

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

# import modules
import os
import numpy as np
import pandas as pd

# functions
def detect_encoding(file_path):
    """
    Detect the encoding of a file.
    """
    import chardet

    with open(file_path, 'rb') as file:
        rawdata = file.read()
    result = chardet.detect(rawdata)
    encoding = result['encoding']
    return encoding

def turn_to_numeric(data_npy):
    ## convert to numeric or NaN
    cleaned_npy = np.full_like(data_npy, fill_value=np.nan, dtype='float')

    for idx in np.ndindex(data_npy.shape): # Iterate through all indexes
        val = data_npy[idx]
        try:
            if val is None:
                cleaned_npy[idx] = np.nan
            else:
                cleaned_npy[idx] = float(val)
        except (ValueError, TypeError):
            cleaned_npy[idx] = np.nan
    
    return cleaned_npy

def print_nan_count(cleaned_npy):
    ## output number of NaNs per monitoring station
    for pollutant, station, in np.ndindex(cleaned_npy.shape[0:2]):
        print(f"Pollutant: {pollutant}, Station: {station}, NaN: {np.isnan(cleaned_npy[pollutant][station]).sum()}, ")

Data processing

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# main programs
## file name
file_name = '即時值查詢.csv'

## EDA
data = pd.read_csv(filepath_or_buffer = file_name,
                   skiprows = 2,
                   encoding = detect_encoding(file_name))
print(data.head())
print(f"data shape: {data.shape}")
Execution result reference
 1
 2
 3
 4
 5
 6
 7
 8
 9
10

   測站          日期  測項    00    01    02    03    04    05    06    07    08  ...    12    13    14    15    16    17    18    19    20    21    22    23
0  基隆  2024/01/01  CO  0.35  0.37  0.38  0.39  0.39  0.39  0.41   0.4  0.39  ...  0.37  0.38  0.36  0.35  0.33  0.32  0.31  0.29  0.29  0.34  0.36  0.39 
1  基隆  2024/01/02  CO  0.39  0.32  0.31  0.32   0.3  0.33  0.48  0.92  1.29  ...  0.29   0.3  0.28  0.29  0.38  0.75  0.75  0.87  0.71  0.58  0.62  0.54 
2  基隆  2024/01/03  CO  0.51  0.58  0.58   0.6  0.61  0.69  0.31  0.31  0.45  ...  0.47     #     #  0.47  0.44  0.43  0.36  0.36  0.33  0.35  0.35  0.34 
3  基隆  2024/01/04  CO  0.33  0.29  0.28  0.33  0.32  0.29  0.29  0.36  0.37  ...  0.31   0.3  0.34  0.32  0.44  0.56   0.8  0.98  1.01  0.76  0.61  0.67 
4  基隆  2024/01/05  CO  0.59  0.45  0.54  0.41  0.38  0.42  0.55  0.62  0.49  ...  0.35  0.35  0.35  0.34  0.37  0.36  0.58  0.62  0.62   0.6  0.54  0.62 

[5 rows x 27 columns]

data shape: (60290, 27)

Data classification

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

## data classification
stations = pd.unique(data.iloc[:, 0])
dates = sorted(pd.unique(data.iloc[:, 1]))
pollutants = pd.unique(data.iloc[:, 2])
print(f"length of data:\n  stations: {len(stations)}, dates: {len(dates)}, total times: {len(dates) * 24}, pollutants: {len(pollutants)}")

data_dict = {pollutant: {station: {date: [None for _ in range(24)] for date in dates} for station in stations} for pollutant in pollutants}

for i in range(data.shape[0]):
    station = data.iloc[i, 0]
    date = data.iloc[i, 1]
    pollutant = data.iloc[i, 2]
    data_dict[pollutant][station][date] = data.iloc[i, 3:27].values
    
## view the first five entries of a specific item
pollutant = pollutants[0]
print(f"Pollutant: '{pollutant}'")
for station, values in data_dict[pollutant].items():
    print(f"{station}: {values[dates[0]][:5]}")
Execution result reference
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

length of data:
  stations: 26, dates: 464, total times: 11136, pollutants: 5

基隆: ['0.35' '0.37' '0.38' '0.39' '0.39']
士林: ['0.35' '0.37' '0.37' '0.39' '0.4']
大同: ['0.48' '0.56' '0.47' '0.46' '0.45']
中山: ['0.41' '0.43' '0.42' '0.42' '0.42']
古亭: ['0.37' '0.35' '0.35' '0.37' '0.39']
松山: ['0.36' '0.37' '0.38' '0.37' '0.38']
陽明: ['0.36' '0.38' '0.39' '0.41' '0.42']
萬華: ['0.4' '0.43' '0.43' '0.42' '0.43']
三重: ['0.63' '0.72' '0.62' '0.6' '0.54']
土城: ['0.36' '0.37' '0.37' '0.37' '0.39']
永和: ['0.43' '0.4' '0.4' '0.4' '0.42']
汐止: ['0.32' '0.35' '0.35' '0.36' '0.37']
板橋: ['0.4' '0.41' '0.4' '0.41' '0.42']
林口: ['0.36' '0.37' '0.38' '0.38' '0.39']
淡水: ['0.33' '0.35' '0.37' '0.38' '0.39']
菜寮: ['0.4' '0.39' '0.4' '0.41' '0.42']
新店: ['0.35' '0.35' '0.35' '0.36' '0.37']
新莊: ['0.34' '0.36' '0.37' '0.37' '0.39']
萬里: ['0.36' '0.38' '0.39' '0.4' '0.4']
富貴角: ['0.31' '0.31' '0.33' '0.34' '0.35']
大園: ['0.35' '0.35' '0.37' '0.38' '0.4']
中壢: ['0.44' '0.47' '0.44' '0.46' '0.43']
平鎮: ['0.36' '0.39' '0.4' '0.41' '0.42']
桃園: ['0.35' '0.37' '0.4' '0.39' '0.4']
龍潭: ['0.36' '0.33' '0.33' '0.34' '0.35']
觀音: ['0.3' '0.33' '0.35' '0.35' '0.36']

Convert file

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

## convert to np
data_npy = []
for pollutant in pollutants:
    station_list = []
    for station in stations:
        time_series = []
        for date in dates:
            time_series.extend(data_dict[pollutant][station][date])
        station_list.append(time_series)
    data_npy.append(station_list)

data_npy = np.asarray(data_npy)
print(data_npy[:, :5, :5])
print(f"data_npy shape: {data_npy.shape}")
Execution result reference
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

[[['0.35' '0.37' '0.38' '0.39' '0.39']
  ['0.35' '0.37' '0.37' '0.39' '0.4']
  ['0.48' '0.56' '0.47' '0.46' '0.45']
  ['0.41' '0.43' '0.42' '0.42' '0.42']
  ['0.37' '0.35' '0.35' '0.37' '0.39']]

 [['4.2' '3.7' '3.5' '3.8' '3.4']
  ['5.3' '5.5' '5' '4.7' '4.5']
  ['10.9' '12.9' '10.3' '8.8' '8.5']
  ['8.5' '9.2' '7.6' '6.4' '6.8']
  ['7.7' '7.5' '5.9' '5.4' '5.7']]

 [['52' '51' '49' '47' '47']
  ['52' '51' '49' '48' '47']
  ['45' '42' '43' '42' '41']
  ['49' '46' '46' '45' '43']
  ['48' '46' '47' '46' '43']]

 [['38' '36' '35' '36' '36']
  ['35' '34' '32' '30' '34']
  ['34' '33' '40' '43' '38']
  ['38' '29' '32' '35' '36']
  ['58' '36' '32' '32' '35']]

 [['1.4' '2.8' '2.3' '2.3' '2.6']
  ['1.3' '1' '0.8' '0.8' '0.9']
  ['1.1' '0.8' '0.8' '0.9' '0.8']
  ['0.7' '1' '0.9' '0.9' '1']
  ['2.1' '1.7' '0.9' '1.1' '1.2']]]

data_npy shape: (5, 26, 11136)

Save file

1
2
3
4
5
6
7
8

## save
np.save('data.npy', data_npy)

## convert to numeric or NaN
cleaned_npy = turn_to_numeric(data_npy)

## output number of NaNs per monitoring station
print_nan_count(cleaned_npy)
Execution result reference
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130

Pollutant: 0, Station: 0, NaN: 176,
Pollutant: 0, Station: 1, NaN: 174,
Pollutant: 0, Station: 2, NaN: 129,
Pollutant: 0, Station: 3, NaN: 117,
Pollutant: 0, Station: 4, NaN: 133,
Pollutant: 0, Station: 5, NaN: 131,
Pollutant: 0, Station: 6, NaN: 95,
Pollutant: 0, Station: 7, NaN: 171,
Pollutant: 0, Station: 8, NaN: 85,
Pollutant: 0, Station: 9, NaN: 120,
Pollutant: 0, Station: 10, NaN: 125,
Pollutant: 0, Station: 11, NaN: 99,
Pollutant: 0, Station: 12, NaN: 96,
Pollutant: 0, Station: 13, NaN: 135,
Pollutant: 0, Station: 14, NaN: 142,
Pollutant: 0, Station: 15, NaN: 90,
Pollutant: 0, Station: 16, NaN: 119,
Pollutant: 0, Station: 17, NaN: 104,
Pollutant: 0, Station: 18, NaN: 165,
Pollutant: 0, Station: 19, NaN: 293,
Pollutant: 0, Station: 20, NaN: 120,
Pollutant: 0, Station: 21, NaN: 127,
Pollutant: 0, Station: 22, NaN: 106,
Pollutant: 0, Station: 23, NaN: 94,
Pollutant: 0, Station: 24, NaN: 134,
Pollutant: 0, Station: 25, NaN: 190,
Pollutant: 1, Station: 0, NaN: 210,
Pollutant: 1, Station: 1, NaN: 222,
Pollutant: 1, Station: 2, NaN: 201,
Pollutant: 1, Station: 3, NaN: 212,
Pollutant: 1, Station: 4, NaN: 199,
Pollutant: 1, Station: 5, NaN: 161,
Pollutant: 1, Station: 6, NaN: 140,
Pollutant: 1, Station: 7, NaN: 216,
Pollutant: 1, Station: 8, NaN: 132,
Pollutant: 1, Station: 9, NaN: 167,
Pollutant: 1, Station: 10, NaN: 169,
Pollutant: 1, Station: 11, NaN: 131,
Pollutant: 1, Station: 12, NaN: 159,
Pollutant: 1, Station: 13, NaN: 198,
Pollutant: 1, Station: 14, NaN: 195,
Pollutant: 1, Station: 15, NaN: 179,
Pollutant: 1, Station: 16, NaN: 182,
Pollutant: 1, Station: 17, NaN: 174,
Pollutant: 1, Station: 18, NaN: 142,
Pollutant: 1, Station: 19, NaN: 432,
Pollutant: 1, Station: 20, NaN: 210,
Pollutant: 1, Station: 21, NaN: 179,
Pollutant: 1, Station: 22, NaN: 248,
Pollutant: 1, Station: 23, NaN: 165,
Pollutant: 1, Station: 24, NaN: 193,
Pollutant: 1, Station: 25, NaN: 261,
Pollutant: 2, Station: 0, NaN: 213,
Pollutant: 2, Station: 1, NaN: 189,
Pollutant: 2, Station: 2, NaN: 244,
Pollutant: 2, Station: 3, NaN: 139,
Pollutant: 2, Station: 4, NaN: 141,
Pollutant: 2, Station: 5, NaN: 169,
Pollutant: 2, Station: 6, NaN: 124,
Pollutant: 2, Station: 7, NaN: 232,
Pollutant: 2, Station: 8, NaN: 151,
Pollutant: 2, Station: 9, NaN: 152,
Pollutant: 2, Station: 10, NaN: 166,
Pollutant: 2, Station: 11, NaN: 241,
Pollutant: 2, Station: 12, NaN: 198,
Pollutant: 2, Station: 13, NaN: 159,
Pollutant: 2, Station: 14, NaN: 186,
Pollutant: 2, Station: 15, NaN: 120,
Pollutant: 2, Station: 16, NaN: 156,
Pollutant: 2, Station: 17, NaN: 157,
Pollutant: 2, Station: 18, NaN: 159,
Pollutant: 2, Station: 19, NaN: 319,
Pollutant: 2, Station: 20, NaN: 154,
Pollutant: 2, Station: 21, NaN: 126,
Pollutant: 2, Station: 22, NaN: 138,
Pollutant: 2, Station: 23, NaN: 124,
Pollutant: 2, Station: 24, NaN: 150,
Pollutant: 2, Station: 25, NaN: 181,
Pollutant: 3, Station: 0, NaN: 174,
Pollutant: 3, Station: 1, NaN: 132,
Pollutant: 3, Station: 2, NaN: 113,
Pollutant: 3, Station: 3, NaN: 116,
Pollutant: 3, Station: 4, NaN: 103,
Pollutant: 3, Station: 5, NaN: 129,
Pollutant: 3, Station: 6, NaN: 155,
Pollutant: 3, Station: 7, NaN: 161,
Pollutant: 3, Station: 8, NaN: 80,
Pollutant: 3, Station: 9, NaN: 117,
Pollutant: 3, Station: 10, NaN: 91,
Pollutant: 3, Station: 11, NaN: 131,
Pollutant: 3, Station: 12, NaN: 92,
Pollutant: 3, Station: 13, NaN: 139,
Pollutant: 3, Station: 14, NaN: 181,
Pollutant: 3, Station: 15, NaN: 77,
Pollutant: 3, Station: 16, NaN: 102,
Pollutant: 3, Station: 17, NaN: 135,
Pollutant: 3, Station: 18, NaN: 147,
Pollutant: 3, Station: 19, NaN: 259,
Pollutant: 3, Station: 20, NaN: 155,
Pollutant: 3, Station: 21, NaN: 138,
Pollutant: 3, Station: 22, NaN: 111,
Pollutant: 3, Station: 23, NaN: 143,
Pollutant: 3, Station: 24, NaN: 126,
Pollutant: 3, Station: 25, NaN: 230,
Pollutant: 4, Station: 0, NaN: 414,
Pollutant: 4, Station: 1, NaN: 260,
Pollutant: 4, Station: 2, NaN: 192,
Pollutant: 4, Station: 3, NaN: 215,
Pollutant: 4, Station: 4, NaN: 501,
Pollutant: 4, Station: 5, NaN: 458,
Pollutant: 4, Station: 6, NaN: 179,
Pollutant: 4, Station: 7, NaN: 374,
Pollutant: 4, Station: 8, NaN: 119,
Pollutant: 4, Station: 9, NaN: 298,
Pollutant: 4, Station: 10, NaN: 156,
Pollutant: 4, Station: 11, NaN: 231,
Pollutant: 4, Station: 12, NaN: 149,
Pollutant: 4, Station: 13, NaN: 345,
Pollutant: 4, Station: 14, NaN: 314,
Pollutant: 4, Station: 15, NaN: 114,
Pollutant: 4, Station: 16, NaN: 182,
Pollutant: 4, Station: 17, NaN: 193,
Pollutant: 4, Station: 18, NaN: 164,
Pollutant: 4, Station: 19, NaN: 485,
Pollutant: 4, Station: 20, NaN: 147,
Pollutant: 4, Station: 21, NaN: 509,
Pollutant: 4, Station: 22, NaN: 366,
Pollutant: 4, Station: 23, NaN: 122,
Pollutant: 4, Station: 24, NaN: 246,
Pollutant: 4, Station: 25, NaN: 222,

Sanyi Data

The following is the code used to process the data from Sanyi Township, with some parts of the code applying previously defined functions.

Data processing

1
2
3
4
5
6
7
8
9

## file name
file_name = 'SanYi_training.csv'

## EDA
data = pd.read_csv(filepath_or_buffer = file_name,
                   skiprows = 0,
                   encoding = detect_encoding(file_name))
print(data.head())
print(f"data shape: {data.shape}")
Execution result reference
1
2
3
4
5
6
7
8

   PM25  PM10    CO    NO    NO2    NOx    O3  PH_RAIN  RAINFALL     RH  SO2   TEMP  UVB  WIND_DIREC  WIND_SPEED
0    20  35.0  0.12  1.21  13.27  14.48  22.8      0.0       0.0  90.51  1.5  14.77  0.0       33.16        4.24
1    16  28.0  0.11  0.98   9.17  10.15  25.7      0.0       0.0  90.25  1.4  14.91  0.0       30.16        4.36
2    16  26.0  0.33  0.78   7.98   8.76  26.9      0.0       0.0  90.24  1.5  15.03  0.0       35.15        4.27
3    15  33.0  0.32  1.08  10.51  11.58  25.4      0.0       0.0  90.41  2.3  14.95  0.0       35.30        4.01
4    13  30.0  0.32  1.07  15.44  16.51  16.4      0.0       0.0  91.48  2.4  14.72  0.0       34.57        4.13

data shape: (43752, 15)

Data classification

1
2
3
4
5
6
7
8

## data classification
print(f"length of data:\n  stations: 1, total times: {data.shape[0]}, pollutants: {data.shape[1]}")
pollutants = data.keys()

data_dict = {pollutant: {'station': {'date': None}} for pollutant in pollutants}

for i in range(data.shape[1]):
    data_dict[data.keys()[i]]['station']['date'] = data.iloc[:, i].values
Execution result reference
1
2

length of data:
  stations: 1, total times: 43752, pollutants: 15

Convert file

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

## convert to np
data_npy = []
for pollutant in pollutants:
    station_list = []
    for station in range(1):
        time_series = []
        for date in range(1):
            time_series.extend(data_dict[pollutant]['station']['date'])
        station_list.append(time_series)
    data_npy.append(station_list)

data_npy = np.asarray(data_npy)
print(data_npy[:, :5, :5])
print(f"data_npy shape: {data_npy.shape}")
Execution result reference
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

[[[20.   16.   16.   15.   13.  ]]

 [[35.   28.   26.   33.   30.  ]]

 [[ 0.12  0.11  0.33  0.32  0.32]]

 [[ 1.21  0.98  0.78  1.08  1.07]]

 [[13.27  9.17  7.98 10.51 15.44]]

 [[14.48 10.15  8.76 11.58 16.51]]

 [[22.8  25.7  26.9  25.4  16.4 ]]

 [[ 0.    0.    0.    0.    0.  ]]

 [[ 0.    0.    0.    0.    0.  ]]

 [[90.51 90.25 90.24 90.41 91.48]]

 [[ 1.5   1.4   1.5   2.3   2.4 ]]

 [[14.77 14.91 15.03 14.95 14.72]]

 [[ 0.    0.    0.    0.    0.  ]]

 [[33.16 30.16 35.15 35.3  34.57]]

 [[ 4.24  4.36  4.27  4.01  4.13]]]

data_npy shape: (15, 1, 43752)

Save file

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

## save
np.save('SanYi.npy', data_npy)

## convert to numeric or NaN
cleaned_npy = turn_to_numeric(data_npy)

## output number of NaNs per monitoring station
print_nan_count(cleaned_npy)

## split into training and testing sets
train_npy = cleaned_npy[:, :, -105:-5]
test_npy = cleaned_npy[:, :, -5:]

train_npy = train_npy.transpose(1, 2, 0)
test_npy = test_npy.transpose(1, 2, 0)
print(train_npy.shape)
print(test_npy.shape)

np.save('SanYi_train.npy', train_npy)
np.save('SanYi_test.npy', test_npy)
Execution result reference
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

Pollutant: 0, Station: 0, NaN: 0,
Pollutant: 1, Station: 0, NaN: 5,
Pollutant: 2, Station: 0, NaN: 0,
Pollutant: 3, Station: 0, NaN: 0,
Pollutant: 4, Station: 0, NaN: 0,
Pollutant: 5, Station: 0, NaN: 0,
Pollutant: 6, Station: 0, NaN: 0,
Pollutant: 7, Station: 0, NaN: 0,
Pollutant: 8, Station: 0, NaN: 0,
Pollutant: 9, Station: 0, NaN: 0,
Pollutant: 10, Station: 0, NaN: 0,
Pollutant: 11, Station: 0, NaN: 0,
Pollutant: 12, Station: 0, NaN: 0,
Pollutant: 13, Station: 0, NaN: 0,
Pollutant: 14, Station: 0, NaN: 0,

(1, 100, 15)

(1, 5, 15)

Taiwan AI Cloud (TWCC) Computing

Config

The lab network went down on April 21st, 2025, and could not connect. Results from April 22nd, 2025.

SSSD_CP/configs/model.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

wavenet:
  # WaveNet model parameters
  input_channels: 15  # Number of input channels
  output_channels: 15  # Number of output channels
  residual_layers: 64  # Number of residual layers
  residual_channels: 256  # Number of channels in residual blocks
  skip_channels: 256  # Number of channels in skip connections

  # Diffusion step embedding dimensions
  diffusion_step_embed_dim_input: 128  # Input dimension
  diffusion_step_embed_dim_hidden: 512  # Middle dimension
  diffusion_step_embed_dim_output: 512  # Output dimension

  # Structured State Spaces sequence model (S4) configurations
  s4_max_sequence_length: 500  # Maximum sequence length
  s4_state_dim: 64  # State dimension
  s4_dropout: 0.0  # Dropout rate
  s4_bidirectional: true  # Whether to use bidirectional layers
  s4_use_layer_norm: true  # Whether to use layer normalization

diffusion:
  # Diffusion model parameters
  T: 200  # Number of diffusion steps
  beta_0: 0.0001  # Initial beta value
  beta_T: 0.02  # Final beta value

SSSD_CP/configs/training.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

# Training configuration
batch_size: 80  # Batch size
output_directory: "./results/checkpoint"  # Output directory for checkpoints and logs
ckpt_iter: "max"  # Checkpoint mode (max or min)
iters_per_ckpt: 100  # Checkpoint frequency (number of epochs)
iters_per_logging: 100  # Log frequency (number of iterations)
n_iters: 5000  # Maximum number of iterations
learning_rate: 0.0002  # Learning rate

# Additional training settings
only_generate_missing: true  # Generate missing values only
use_model: 2  # Model to use for training
masking: "forecast"  # Masking strategy for missing values
missing_k: 5  # Number of missing values

# Data paths
data:
  train_path: "./datasets/SanYi/SanYi_train.npy"  # Path to training data

SSSD_CP/configs/inference.yaml

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

# Inference configuration
batch_size: 80  # Batch size for inference
output_directory: "./results/checkpoint"  # Output directory for inference results
ckpt_path: "./results/checkpoint"  # Path to checkpoint for inference
trials: 1 # Replications

# Additional training settings
only_generate_missing: true  # Generate missing values only
use_model: 2  # Model to use for training
masking: "forecast"  # Masking strategy for missing values
missing_k: 5  # Number of missing values

# Data paths
data:
  test_path: "./datasets/SanYi/SanYi_test.npy"  # Path to test data

Commands Used on TWCC

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43

# Query current directory
pwd

# Download Miniconda
wget -c https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh

# Install Miniconda
bash Miniconda3-latest-Linux-x86_64.sh -b -f -p /home/u6025091

# Check conda version (verify installation)
conda --version

# Create SSSD environment using Python 3.10
conda create -n sssd python=3.10 -y

# Install dos2unix, used for converting line endings
sudo apt-get update
sudo apt-get install dos2unix

# Convert all files under `/SSSD_CP/`
find ~/SSSD_CP/ -type f -exec dos2unix {} \;

# Elevate execution permissions
./envs/vm/install_on_ubuntu.sh
./envs/conda/build_conda_env.sh

chmod +x /SSSD_CP/scripts/diffusion/training_job.sh

# Install libcudnn.so.8 (if error occurs stating it can't be found)
sudo apt-get update
sudo apt-get install libcudnn8 libcudnn8-dev libcudnn8-doc

# Find the location of libcudnn.so.8
find / -name "libcudnn.so.8" 2>/dev/null

# Add libcudnn.so.8 to environment variable (use location found above)
export LD_LIBRARY_PATH=/home/u6025091/local/envs/sssd/lib/python3.10/site-packages/nvidia/cudnn/lib:$LD_LIBRARY_PATH

# Reload environment variables
source ~/.bashrc

# Run model training
./scripts/diffusion/training_job.sh -m configs/model.yaml -t configs/training.yaml

Computational Results

The lab network went down on April 21st, 2025, and could not connect. Results from April 22nd, 2025.
Execution result reference
  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438

(base) u6025091@jjd26etest1140422-b452q:~$ s3cmd get --recursive --verbose s3://sssds4/SSSD_CP/datasets/SanYi/ ~/SSSD_CP/datasets/SanYi/
INFO: Retrieving list of remote files for s3://sssds4/SSSD_CP/datasets/SanYi/ ...
WARNING: Found empty root object name on S3, ignoring.
INFO: Summary: 2 remote files to download
ERROR: Parameter problem: Destination must be a directory or stdout when downloading multiple sources.
(base) u6025091@jjd26etest1140422-b452q:~$ pwd
/home/u6025091
(base) u6025091@jjd26etest1140422-b452q:~$ s3cmd get --recursive --verbose s3://sssds4/SSSD_CP/datasets/SanYi/ ~/SSSD_CP/datasets/SanYi/
INFO: Retrieving list of remote files for s3://sssds4/SSSD_CP/datasets/SanYi/ ...
WARNING: Found empty root object name on S3, ignoring.
INFO: Summary: 2 remote files to download
ERROR: Parameter problem: Destination must be a directory or stdout when downloading multiple sources.
(base) u6025091@jjd26etest1140422-b452q:~$ s3cmd get --recursive --verbose s3://sssds4/SSSD_CP/datasets/SanYi/* ~/SSSD_CP/datasets/SanYi/
INFO: Retrieving list of remote files for s3://sssds4/SSSD_CP/datasets/SanYi/* ...
INFO: Summary: 0 remote files to download
(base) u6025091@jjd26etest1140422-b452q:~$ s3cmd get --recursive --verbose s3://sssds4/SSSD_CP/datasets/SanYi/*.npy ~/SSSD_CP/datasets/SanYi/
INFO: Retrieving list of remote files for s3://sssds4/SSSD_CP/datasets/SanYi/*.npy ...
INFO: Summary: 0 remote files to download
(base) u6025091@jjd26etest1140422-b452q:~$ s3cmd get --recursive --verbose s3://sssds4/SSSD_CP/datasets/SanYi/ ~/SSSD_CP/datasets/SanYi/
INFO: Retrieving list of remote files for s3://sssds4/SSSD_CP/datasets/SanYi/ ...
WARNING: Found empty root object name on S3, ignoring.
INFO: Summary: 2 remote files to download
ERROR: Parameter problem: Destination must be a directory or stdout when downloading multiple sources.
(base) u6025091@jjd26etest1140422-b452q:~$ s3cmd get --recursive --verbose s3://sssds4/SSSD_CP/datasets/SanYi/ ~/SSSD_CP/datasets/SanYi/
INFO: Retrieving list of remote files for s3://sssds4/SSSD_CP/datasets/SanYi/ ...
WARNING: Found empty root object name on S3, ignoring.
INFO: Summary: 2 remote files to download
download: 's3://sssds4/SSSD_CP/datasets/SanYi/SanYi_test.npy' -> '/home/u6025091/SSSD_CP/datasets/SanYi/SanYi_test.npy'  [1 of 2]
 12728 of 12728   100% in    0s    53.88 KB/s  done
download: 's3://sssds4/SSSD_CP/datasets/SanYi/SanYi_train.npy' -> '/home/u6025091/SSSD_CP/datasets/SanYi/SanYi_train.npy'  [2 of 2]
 60128 of 60128   100% in    0s   222.61 KB/s  done
(base) u6025091@jjd26etest1140422-b452q:~$ conda envs list
usage: conda [-h] [-v] [--no-plugins] [-V] COMMAND ...
conda: error: argument COMMAND: invalid choice: 'envs' (choose from activate, clean, commands, compare, config, content-trust, create, deactivate, doctor, env, export, info, init, install, list, notices, package, remove, rename, repoquery, run, search, tos, uninstall, update, upgrade)
(base) u6025091@jjd26etest1140422-b452q:~$ conda env list

# conda environments:
#
                       /home/u6025091
base                 * /home/u6025091/local
sssd                   /home/u6025091/local/envs/sssd

(base) u6025091@jjd26etest1140422-b452q:~$ deactivate
DeprecationWarning: 'source deactivate' is deprecated. Use 'conda deactivate'.
(base) u6025091@jjd26etest1140422-b452q:~$ conda deactivate
u6025091@jjd26etest1140422-b452q:~$ conda activate sssd
(sssd) u6025091@jjd26etest1140422-b452q:~$ ./scripts/diffusion/training_job.sh -m configs/model.yaml -t configs/training.yaml
bash: ./scripts/diffusion/training_job.sh: No such file or directory
(sssd) u6025091@jjd26etest1140422-b452q:~$ pwd
/home/u6025091
(sssd) u6025091@jjd26etest1140422-b452q:~$ cd SSSD_CP
(sssd) u6025091@jjd26etest1140422-b452q:~/SSSD_CP$ ./scripts/diffusion/training_job.sh -m configs/model.yaml -t configs/training.yaml
Intializing conda
Activating Conda Env: sssd
[Execution - Training]
/home/u6025091/SSSD_CP/scripts/diffusion/train.py --model_config configs/model.yaml --training_config configs/training.yaml
Traceback (most recent call last):
  File "/home/u6025091/SSSD_CP/scripts/diffusion/train.py", line 5, in <module>
    import torch
  File "/home/u6025091/.local/lib/python3.10/site-packages/torch/__init__.py", line 237, in <module>
    from torch._C import *  # noqa: F403
ImportError: libcudnn.so.8: cannot open shared object file: No such file or directory
(sssd) u6025091@jjd26etest1140422-b452q:~/SSSD_CP$ find ~/SSSD_CP/ -type f -exec dos2unix {} \;
dos2unix: converting file /home/u6025091/SSSD_CP/tests/inference/test_inference_utils.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/tests/core/__init__.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/tests/core/layers/test_layers_activation.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/tests/core/layers/__init__.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/tests/core/layers/test_layers_linear.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/tests/core/hippo/__init__.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/tests/core/hippo/test_hippo_utils.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/tests/core/hippo/test_hippo_state_space.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/tests/core/hippo/test_normal_plus_low_rank.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/tests/core/hippo/test_hippo_kernel.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/tests/core/test_core_utils.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/tests/__init__.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/tests/utils/test_utils_utils.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/tests/utils/__init__.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/tests/utils/test_logger.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/tests/data/test_data_generator.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/tests/data/__init__.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/tests/data/test_data_dataloader.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/tests/data/test_data_utils.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/tests/data/test_data_dataset.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/tests/training/test_training_utils.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/tests/training/__init__.py to Unix format...
dos2unix: Binary symbol 0x1A found at line 2
dos2unix: Skipping binary file /home/u6025091/SSSD_CP/docs/report/plots_merged_004.png
dos2unix: Binary symbol 0x1A found at line 2
dos2unix: Skipping binary file /home/u6025091/SSSD_CP/docs/report/plots_merged_001.png
dos2unix: converting file /home/u6025091/SSSD_CP/docs/report/sssd.drawio_bigger.xml to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/docs/report/sssd.drawio to Unix format...
dos2unix: Binary symbol 0x1C found at line 174
dos2unix: Skipping binary file /home/u6025091/SSSD_CP/docs/report/sssd.drawio_bigger.pdf
dos2unix: Binary symbol 0x1A found at line 2
dos2unix: Skipping binary file /home/u6025091/SSSD_CP/docs/report/plots_merged_002.png
dos2unix: converting file /home/u6025091/SSSD_CP/docs/report/SSSDS4draw.xml to Unix format...
dos2unix: Binary symbol 0x1A found at line 2
dos2unix: Skipping binary file /home/u6025091/SSSD_CP/docs/report/plots_merged_003.png
dos2unix: Binary symbol 0x1A found at line 2
dos2unix: Skipping binary file /home/u6025091/SSSD_CP/docs/report/updated_architecture.png
dos2unix: Binary symbol 0x1A found at line 2
dos2unix: Skipping binary file /home/u6025091/SSSD_CP/docs/data_plt/uncleaned_20050131_20161231/HUD VL.png
dos2unix: Binary symbol 0x1A found at line 2
dos2unix: Skipping binary file /home/u6025091/SSSD_CP/docs/data_plt/uncleaned_20050131_20161231/DUNWOD.png
dos2unix: Binary symbol 0x1A found at line 2
dos2unix: Skipping binary file /home/u6025091/SSSD_CP/docs/data_plt/uncleaned_20050131_20161231/CAPITL.png
dos2unix: Binary symbol 0x1A found at line 2
dos2unix: Skipping binary file /home/u6025091/SSSD_CP/docs/data_plt/uncleaned_20050131_20161231/GENESE.png
dos2unix: Binary symbol 0x1A found at line 2
dos2unix: Skipping binary file /home/u6025091/SSSD_CP/docs/data_plt/uncleaned_20050131_20161231/NORTH.png
dos2unix: Binary symbol 0x1A found at line 2
dos2unix: Skipping binary file /home/u6025091/SSSD_CP/docs/data_plt/uncleaned_20050131_20161231/CENTRL.png
dos2unix: Binary symbol 0x1A found at line 2
dos2unix: Skipping binary file /home/u6025091/SSSD_CP/docs/data_plt/uncleaned_20050131_20161231/LONGIL.png
dos2unix: Binary symbol 0x1A found at line 2
dos2unix: Skipping binary file /home/u6025091/SSSD_CP/docs/data_plt/uncleaned_20050131_20161231/MILLWD.png
dos2unix: Binary symbol 0x1A found at line 2
dos2unix: Skipping binary file /home/u6025091/SSSD_CP/docs/data_plt/uncleaned_20050131_20161231/MHK VL.png
dos2unix: Binary symbol 0x1A found at line 2
dos2unix: Skipping binary file /home/u6025091/SSSD_CP/docs/data_plt/uncleaned_20050131_20161231/N.Y.C..png
dos2unix: Binary symbol 0x1A found at line 2
dos2unix: Skipping binary file /home/u6025091/SSSD_CP/docs/data_plt/uncleaned_20050131_20161231/WEST.png
dos2unix: converting file /home/u6025091/SSSD_CP/docs/instructions/Solar/solar.ipynb to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/docs/instructions/Solar/updated_solar.ipynb to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/docs/instructions/ETTm1/ettm1.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/docs/instructions/PEMS-Bay and METR-LA/README.md to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/docs/instructions/PEMS-Bay and METR-LA/feature_sample_process.ipynb to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/docs/instructions/PTB-XL/ecg_data_preprocessing.ipynb to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/docs/instructions/PTB-XL/clinical_ts/ecg_utils.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/docs/instructions/PTB-XL/clinical_ts/timeseries_utils.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/docs/instructions/PTB-XL/README.md to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/docs/instructions/Electricity/README.md to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/docs/instructions/Mujoco/README.md to Unix format...
dos2unix: Binary symbol 0x1A found at line 2
dos2unix: Skipping binary file /home/u6025091/SSSD_CP/docs/images/docker_compose_ps.png
dos2unix: Binary symbol 0x1A found at line 2
dos2unix: Skipping binary file /home/u6025091/SSSD_CP/docs/images/docker_compose_down.png
dos2unix: Binary symbol 0x1A found at line 2
dos2unix: Skipping binary file /home/u6025091/SSSD_CP/docs/images/run_docker.png
dos2unix: Binary symbol 0x1A found at line 2
dos2unix: Skipping binary file /home/u6025091/SSSD_CP/docs/images/clean_docker.png
dos2unix: Binary symbol 0x1A found at line 2
dos2unix: Skipping binary file /home/u6025091/SSSD_CP/docs/images/img_docker_compose_log.png
dos2unix: Binary symbol 0x01 found at line 1
dos2unix: Skipping binary file /home/u6025091/SSSD_CP/datasets/SanYi/SanYi_test.npy
dos2unix: Binary symbol 0x01 found at line 1
dos2unix: Skipping binary file /home/u6025091/SSSD_CP/datasets/SanYi/SanYi_train.npy
dos2unix: converting file /home/u6025091/SSSD_CP/.pre-commit-config.yaml to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/bin/utils.sh to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/bin/color_map.sh to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/bin/exit_code.sh to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/Makefile to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/.github/workflows/docker.yaml to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/.github/workflows/test.yaml to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/.github/pull_request_template.md to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/sssd/inference/generator.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/sssd/inference/utils.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/sssd/inference/___init__.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/sssd/core/imputers/SSSDSAImputer.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/sssd/core/imputers/SSSDS4Imputer.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/sssd/core/imputers/DiffWaveImputer.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/sssd/core/imputers/___init__.py to Unix format...
dos2unix: Binary symbol 0x00 found at line 2
dos2unix: Skipping binary file /home/u6025091/SSSD_CP/sssd/core/imputers/__pycache__/DiffWaveImputer.cpython-312.pyc
dos2unix: Binary symbol 0x00 found at line 2
dos2unix: Skipping binary file /home/u6025091/SSSD_CP/sssd/core/imputers/__pycache__/SSSDS4Imputer.cpython-312.pyc
dos2unix: Binary symbol 0x00 found at line 2
dos2unix: Skipping binary file /home/u6025091/SSSD_CP/sssd/core/imputers/__pycache__/SSSDSAImputer.cpython-312.pyc
dos2unix: converting file /home/u6025091/SSSD_CP/sssd/core/utils.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/sssd/core/___init__.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/sssd/core/model_specs.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/sssd/core/layers/activation.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/sssd/core/layers/linear.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/sssd/core/layers/___init__.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/sssd/core/layers/s4/s4_layer.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/sssd/core/layers/s4/___init__.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/sssd/core/layers/s4/hippo/hippo.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/sssd/core/layers/s4/hippo/utils.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/sssd/core/layers/s4/hippo/___init__.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/sssd/core/layers/s4/hippo/state_space.py to Unix format...
dos2unix: Binary symbol 0x00 found at line 2
dos2unix: Skipping binary file /home/u6025091/SSSD_CP/sssd/core/layers/s4/hippo/__pycache__/state_space.cpython-312.pyc
dos2unix: Binary symbol 0x00 found at line 2
dos2unix: Skipping binary file /home/u6025091/SSSD_CP/sssd/core/layers/s4/hippo/__pycache__/utils.cpython-312.pyc
dos2unix: Binary symbol 0x00 found at line 2
dos2unix: Skipping binary file /home/u6025091/SSSD_CP/sssd/core/layers/s4/hippo/__pycache__/hippo.cpython-312.pyc
dos2unix: Binary symbol 0x00 found at line 2
dos2unix: Skipping binary file /home/u6025091/SSSD_CP/sssd/core/layers/s4/hippo/__pycache__/normal_plus_low_rank.cpython-312.pyc
dos2unix: converting file /home/u6025091/SSSD_CP/sssd/core/layers/s4/hippo/normal_plus_low_rank.py to Unix format...
dos2unix: Binary symbol 0x00 found at line 2
dos2unix: Skipping binary file /home/u6025091/SSSD_CP/sssd/core/layers/s4/__pycache__/s4_layer.cpython-312.pyc
dos2unix: converting file /home/u6025091/SSSD_CP/sssd/core/layers/layer_normalization.py to Unix format...
dos2unix: Binary symbol 0x00 found at line 2
dos2unix: Skipping binary file /home/u6025091/SSSD_CP/sssd/core/layers/__pycache__/linear.cpython-312.pyc
dos2unix: Binary symbol 0x00 found at line 2
dos2unix: Skipping binary file /home/u6025091/SSSD_CP/sssd/core/layers/__pycache__/activation.cpython-312.pyc
dos2unix: Binary symbol 0x00 found at line 2
dos2unix: Skipping binary file /home/u6025091/SSSD_CP/sssd/core/__pycache__/utils.cpython-312.pyc
dos2unix: Binary symbol 0x00 found at line 2
dos2unix: Skipping binary file /home/u6025091/SSSD_CP/sssd/core/__pycache__/model_specs.cpython-312.pyc
dos2unix: converting file /home/u6025091/SSSD_CP/sssd/utils/utils.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/sssd/utils/___init__.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/sssd/utils/data_utils.py to Unix format...
dos2unix: Binary symbol 0x00 found at line 2
dos2unix: Skipping binary file /home/u6025091/SSSD_CP/sssd/utils/__pycache__/logger.cpython-312.pyc
dos2unix: Binary symbol 0x00 found at line 2
dos2unix: Skipping binary file /home/u6025091/SSSD_CP/sssd/utils/__pycache__/utils.cpython-312.pyc
dos2unix: converting file /home/u6025091/SSSD_CP/sssd/utils/logger.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/sssd/___init__.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/sssd/data/generator.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/sssd/data/dataloader.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/sssd/data/utils.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/sssd/data/___init__.py to Unix format...
dos2unix: Binary symbol 0x00 found at line 2
dos2unix: Skipping binary file /home/u6025091/SSSD_CP/sssd/data/__pycache__/generator.cpython-312.pyc
dos2unix: Binary symbol 0x00 found at line 2
dos2unix: Skipping binary file /home/u6025091/SSSD_CP/sssd/data/__pycache__/utils.cpython-312.pyc
dos2unix: converting file /home/u6025091/SSSD_CP/sssd/data/dataset.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/sssd/training/trainer.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/sssd/training/utils.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/sssd/training/___init__.py to Unix format...
dos2unix: Binary symbol 0x00 found at line 2
dos2unix: Skipping binary file /home/u6025091/SSSD_CP/sssd/training/__pycache__/trainer.cpython-312.pyc
dos2unix: Binary symbol 0x00 found at line 2
dos2unix: Skipping binary file /home/u6025091/SSSD_CP/sssd/training/__pycache__/utils.cpython-312.pyc
dos2unix: converting file /home/u6025091/SSSD_CP/LICENCE to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/notebooks/experiments/MAPE.ipynb to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/notebooks/experiments/datasets/pickle-loadfrst.ipynb to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/notebooks/experiments/datasets/nyiso-csv-to-pickle.ipynb to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/notebooks/experiments/datasets/fake_data.ipynb to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/notebooks/experiments/datasets/vis.ipynb to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/notebooks/experiments/datasets/csv-to-pickle.ipynb to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/notebooks/experiments/datasets/load-pickle-to-npy.ipynb to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/notebooks/experiments/datasets/nyiso-load-pickle-to-npy.ipynb to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/notebooks/experiments/normal_plus_low_rank_measures.ipynb to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/notebooks/experiments/compare_einsum.ipynb to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/notebooks/experiments/ar training test.ipynb to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/notebooks/experiments/vis.ipynb to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/notebooks/experiments/CPI.ipynb to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/notebooks/examples/training.ipynb to Unix format...
dos2unix: Binary symbol 0x00 found at line 5
dos2unix: Skipping binary file /home/u6025091/SSSD_CP/envs/vm/cuda-keyring_1.1-1_all.deb
dos2unix: converting file /home/u6025091/SSSD_CP/envs/vm/install_on_ubuntu.sh to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/envs/vm/.gitignore to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/envs/docker/Dockerfile.full to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/envs/docker/README.md to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/envs/jupyter/utils.sh to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/envs/jupyter/start_jupyter_lab.sh to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/envs/conda/pack_conda_env.sh to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/envs/conda/build_conda_env.sh to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/envs/conda/utils.sh to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/envs/conda/find_conda_env.sh to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/envs/conda/cauchy_mult_setup.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/envs/conda/.gitignore to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/envs/conda/conda_env_info.sh to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/poetry.lock to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/configs/model.yaml to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/configs/.ipynb_checkpoints/README-checkpoint.md to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/configs/inference.yaml to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/configs/training.yaml to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/configs/README.md to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/configs/legacy/config_SSSDS4-NYISO-2-mix-fine-tune-1.json to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/configs/legacy/config_SSSDS4-NYISO.json to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/configs/legacy/config_SSSDS4-NYISO-3-MILLWD.json to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/configs/legacy/config_SSSDS4-NYISO-2-all-fine-tune-1.json to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/configs/legacy/config_SSSDS4-NYISO-3-CAPITL.json to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/configs/legacy/config_SSSDS4-NYISO-split-2.json to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/configs/legacy/config_SSSDS4.json to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/configs/legacy/config_SSSDS4-ar1.json to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/configs/legacy/config_SSSDS4-NYISO-2-all-fine-tune-2.json to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/configs/legacy/config_SSSDSA.json to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/configs/legacy/toy_example.json to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/configs/legacy/config_SSSDS4-NYISO-3-mix.json to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/configs/legacy/config_SSSDS4-NYISO-3-all.json to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/configs/legacy/config_SSSDS4-NYISO-2-mix.json to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/configs/legacy/config_SSSDS4-NYISO-all.json to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/configs/legacy/config_SSSDS4-NYISO-2-all.json to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/configs/legacy/config_SSSDS4-NYISO-split.json to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/configs/legacy/config_DiffWave.json to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/configs/legacy/config_SSSDS4-NYISO-3-NYC.json to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/configs/legacy/config_SSSDS4-NYISO-2-mix-fine-tune-2.json to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/Dockerfile to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/README.md to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/pyproject.toml to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/.gitignore to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/services/jupyter-docker-compose.yaml to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/services/diffusion-docker-compose.yaml to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/scripts/docker/run_diffusion.sh to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/scripts/docker/run_notebook.sh to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/scripts/diffusion/inference_job.sh to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/scripts/diffusion/.ipynb_checkpoints/training_job-checkpoint.sh to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/scripts/diffusion/infer.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/scripts/diffusion/train.py to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/scripts/diffusion/training_job.sh to Unix format...
dos2unix: converting file /home/u6025091/SSSD_CP/scripts/diffusion/diffusion_process.sh to Unix format...
dos2unix: Binary symbol 0x02 found at line 670
dos2unix: Skipping binary file /home/u6025091/SSSD_CP/Miniconda3-latest-Linux-x86_64.sh
(sssd) u6025091@jjd26etest1140422-b452q:~/SSSD_CP$ ./scripts/diffusion/training_job.sh -m configs/model.yaml -t configs/training.yaml
Intializing conda
Activating Conda Env: sssd
[Execution - Training]
/home/u6025091/SSSD_CP/scripts/diffusion/train.py --model_config configs/model.yaml --training_config configs/training.yaml
Traceback (most recent call last):
  File "/home/u6025091/SSSD_CP/scripts/diffusion/train.py", line 5, in <module>
    import torch
  File "/home/u6025091/.local/lib/python3.10/site-packages/torch/__init__.py", line 237, in <module>
    from torch._C import *  # noqa: F403
ImportError: libcudnn.so.8: cannot open shared object file: No such file or directory
(sssd) u6025091@jjd26etest1140422-b452q:~/SSSD_CP$ nvidia-smi
Tue Apr 22 09:07:30 2025       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.161.08             Driver Version: 535.161.08   CUDA Version: 12.8     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla V100-SXM2-32GB           On  | 00000000:1B:00.0 Off |                    0 |
| N/A   26C    P0              42W / 300W |      0MiB / 32768MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+
(sssd) u6025091@jjd26etest1140422-b452q:~/SSSD_CP$ find / -name "libcudnn.so.8" 2>/dev/null
/home/u6025091/local/envs/sssd/lib/python3.10/site-packages/nvidia/cudnn/lib/libcudnn.so.8
(sssd) u6025091@jjd26etest1140422-b452q:~/SSSD_CP$ export LD_LIBRARY_PATH=/home/u6025091/local/envs/sssd/lib/python3.10/site-packages/nvidia/cudnn/lib:$LD_LIBRARY_PATH
(sssd) u6025091@jjd26etest1140422-b452q:~/SSSD_CP$ source ~/.bashrc
(base) u6025091@jjd26etest1140422-b452q:~/SSSD_CP$ ./scripts/diffusion/training_job.sh -m configs/model.yaml -t configs/training.yaml
Intializing conda
Activating Conda Env: sssd
[Execution - Training]
/home/u6025091/SSSD_CP/scripts/diffusion/train.py --model_config configs/model.yaml --training_config configs/training.yaml
2025-04-22 09:08:22,366 - sssd.utils.logger - INFO - Model spec: {'wavenet': {'input_channels': 15, 'output_channels': 15, 'residual_layers': 64, 'residual_channels': 256, 'skip_channels': 256, 'diffusion_step_embed_dim_input': 128, 'diffusion_step_embed_dim_hidden': 512, 'diffusion_step_embed_dim_output': 512, 's4_max_sequence_length': 500, 's4_state_dim': 64, 's4_dropout': 0.0, 's4_bidirectional': True, 's4_use_layer_norm': True}, 'diffusion': {'T': 200, 'beta_0': 0.0001, 'beta_T': 0.02}}
2025-04-22 09:08:22,366 - sssd.utils.logger - INFO - Training spec: {'batch_size': 80, 'output_directory': './results/checkpoint', 'ckpt_iter': 'max', 'iters_per_ckpt': 100, 'iters_per_logging': 100, 'n_iters': 5000, 'learning_rate': 0.0002, 'only_generate_missing': True, 'use_model': 2, 'masking': 'forecast', 'missing_k': 5, 'data': {'train_path': './datasets/SanYi/SanYi_train.npy'}}
2025-04-22 09:08:22,377 - sssd.utils.logger - INFO - Using 1 GPUs!
2025-04-22 09:08:22,409 - sssd.utils.logger - INFO - Output directory ./results/checkpoint/T200_beta00.0001_betaT0.02
2025-04-22 09:08:40,246 - sssd.utils.logger - INFO - Current time: 2025-04-22 09:08:40
2025-04-22 09:08:43,877 - sssd.utils.logger - INFO - No valid checkpoint model found, start training from initialization.
2025-04-22 09:08:43,877 - sssd.utils.logger - INFO - Start the 1 iteration
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:04<00:00,  4.44s/it]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.66it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.82it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.82it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.80it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.82it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.79it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.77it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.82it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.82it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.80it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.82it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.81it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.83it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.82it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.53it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.79it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.81it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.82it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.79it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.81it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.82it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.81it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.83it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.79it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.79it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.82it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.83it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.82it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.80it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.80it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.84it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.82it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.79it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.80it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.82it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.82it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.83it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.82it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.81it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.83it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.82it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.77it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.79it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.83it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.84it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.84it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.82it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.80it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.83it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.82it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.79it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.79it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.83it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.83it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.83it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.84it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.83it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.79it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.83it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.79it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.81it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.83it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.59it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.84it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.82it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.84it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.82it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.80it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.79it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.81it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.84it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.83it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.83it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.83it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.82it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.84it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.82it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.78it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.80it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.83it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.81it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.82it/s]
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.82it/s]





100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  1.83it/s]
2025-04-22 09:56:48,448 - sssd.utils.logger - INFO - Iteration: 5000    Loss: 0.971707284450531
2025-04-22 09:56:49,823 - sssd.utils.logger - INFO - Current time: 2025-04-22 09:56:49
Training Job completed

https://raw.githubusercontent.com/Josh-test-lab/website-assets-repository/refs/heads/main/posts/1140422%20meeting/1745285023058.png
Real-time resource usage.

Environment

  • Local Operating System: Windows 11 24H2
    • Programming Language: Python 3.12.9
  • Computing Platform: National Center for High-Performance Computing (NCHC) – Taiwan AI Cloud
    • Operating System: Ubuntu
    • Miniconda
    • GPU: NVIDIA Tesla V100 32GB GPU
    • CUDA 12.8 driver
    • Programming Language: Python 3.10.16 for Linux

Further Learning

See Also

References

Updated on 2025-04-23 13:09 UTC+0800   6b3838d2
Creative Commons License
 Meeting, $SSSD^{S4}$, Python
Back | Home