1150317 meeting
收尾工作 v2.0
本次實驗原先欲複測 Weather2K 、 NASA GES DISC MERRA-2 在 與上次實驗相同資料範圍下,但不進行時間重塑的實驗結果。但因使用國網中心時,發現已超越單一 GPU 之 vRAM 限制(過大),故本次將對所有實驗進行參數調整,縮小 Residual 層數並重測所有實驗,已達可用之實驗組與對照組。
本次實驗順帶重構以往所有用於實驗之程式碼(SSSDS4 + AFRK 程式碼未更動),同時將兩資料集實驗範圍統一在半年區間,時序預測設為其 10 % ,約 18 天;未觀測地點為整體地點之 20 % 。
在研究為何原 SSSD 程式碼無法直接於多 GPU 上執行運算時,發現其中多步接直接指定計算裝置為 CDUA ,即
| |
此舉造成使用 SSSD 程式碼時,被限定在僅可使用 NVIDIA GPU ,從而無法使用 CPU 、 MPS 、 TPU 等裝置運行,且硬性編碼 .device("cuda") 會造成只能於第一個 GPU ,即 "cuda:0" 上運行,使多 GPU 間無法進行資料傳遞。因整體進行改寫曠日廢時,故目前暫無打算進行改寫,僅以同上所述,縮減部分模型層數以進行實驗。
實驗介紹
本次進行以下實驗,詳細實驗設定參考下節。
| 控制變因 | 實驗 1 | 實驗 2 | 實驗 3 | 實驗 4 |
|---|---|---|---|---|
| 迭代次數 | 4,000 | 4,000 | 4,000 | 4,000 |
| 訓練策略 | $SSSD^{S4 + AFRK}$ | $SSSD^{S4}$ | $SSSD^{S4 + AFRK}$ | $SSSD^{S4}$ |
| 時間重塑 | false | false | true | true |
| 併入時間步 $p$ | — | — | 8 / 24 | 8 / 24 |
| Input Channels | 4 / 1 | 4 / 1 | 32 / 24 | 32 / 24 |
| S4 Max Seq. Length | 1,176 / 4,608 | 1,176 / 4,608 | 147 / 192 | 147 / 192 |
| Missing $k$ | 88 / 144 | 88 / 144 | 11 / 6 | 11 / 6 |
實驗命名依上表不同可區分如下:
- 實驗 1
- Weather2k-1var-S4+AFRK
- MERRA2-1var-S4+AFRK
- 實驗 2
- Weather2k-1var-S4
- MERRA2-1var-S4
- 實驗 3
- Weather2k-S4+AFRK
- MERRA2-S4+AFRK
- 實驗 4
- Weather2k-S4
- MERRA2-S4
實驗設定
本次實驗使用設定如下所示,僅輸入出通道(channels)、時間序列長度(s4 max sequence length)、缺失值(missing $k$)、是否啟用 AFRK(enable spatial training)及目錄與路徑會依實驗內容不同而有更動。以下呈現 Weather2k-S4+AFRK 設定。
model.yaml
| |
training.yaml
| |
inference.yaml
| |
選取的實驗地點如下所示。

下節呈現各實驗結果。
實驗 1
Weather2k-1var-S4+AFRK
訓練花費時間:35h 25m 14.82s(註:此實驗非使用單一機器,而是與 Weather2k-1var-S4 、 Weather2k-S4 、 Weather2k-S4+AFRK 共用機器進行訓練)
推論階段 AFRK 之 MRTS 基底數:382
| Metric | ALL Locs & All Time | Known Locs & All Time | Unknown Locs & All Time | ALL Locs & Future | Known Locs & Future | Unknown Locs & Future | ALL Locs & Past | Known Locs & Past | Unknown Locs & Past |
|---|---|---|---|---|---|---|---|---|---|
| MSPE | 1.494639e+01 | 1.160173e+01 | 2.828929e+01 | 7.756372e+01 | 7.868970e+01 | 7.307184e+01 | 8.156561e+00 | 4.327126e+00 | 2.343335e+01 |
| RMSPE | 3.866056e+00 | 3.406131e+00 | 5.318767e+00 | 8.807027e+00 | 8.870722e+00 | 8.548207e+00 | 2.855969e+00 | 2.080175e+00 | 4.840800e+00 |
| MSPE% | 1.380822e+10 | 3.299012e+09 | 5.573267e+10 | 2.246783e+10 | 2.608424e+10 | 8.040864e+09 | 1.286923e+10 | 8.283242e+08 | 6.090407e+10 |
| RMSPE% | 1.175084e+05 | 5.743703e+04 | 2.360777e+05 | 1.498927e+05 | 1.615062e+05 | 8.967086e+04 | 1.134426e+05 | 2.878062e+04 | 2.467875e+05 |
| MAPE | 2.332151e+00 | 2.120980e+00 | 3.174576e+00 | 5.441194e+00 | 5.450313e+00 | 5.404813e+00 | 1.995026e+00 | 1.759968e+00 | 2.932743e+00 |
| MAPE% | 7.822800e+08 | 5.627048e+08 | 1.658233e+09 | 5.047446e+08 | 5.480241e+08 | 3.320892e+08 | 8.123742e+08 | 5.642966e+08 | 1.802031e+09 |
MERRA2-1var-S4+AFRK
訓練花費時間:20h 24m 12.59s
推論階段 AFRK 之 MRTS 基底數:301
| Metric | ALL Locs & All Time | Known Locs & All Time | Unknown Locs & All Time | ALL Locs & Future | Known Locs & Future | Unknown Locs & Future | ALL Locs & Past | Known Locs & Past | Unknown Locs & Past |
|---|---|---|---|---|---|---|---|---|---|
| MSPE | 160.204191 | 160.390768 | 159.457883 | 1628.647842 | 1631.785612 | 1616.096765 | 0.975361 | 0.841929 | 1.509089 |
| RMSPE | 12.657179 | 12.664548 | 12.627663 | 40.356509 | 40.395366 | 40.200706 | 0.987604 | 0.917567 | 1.228450 |
| MSPE% | 0.611571 | 0.612622 | 0.607367 | 6.222452 | 6.237316 | 6.162993 | 0.003162 | 0.002715 | 0.004950 |
| RMSPE% | 0.782030 | 0.782702 | 0.779338 | 2.494484 | 2.497462 | 2.482538 | 0.056233 | 0.052108 | 0.070357 |
| MAPE | 3.875349 | 3.801459 | 4.170910 | 34.341577 | 34.369929 | 34.228172 | 0.571782 | 0.486805 | 0.911689 |
| MAPE% | 0.014226 | 0.013981 | 0.015206 | 0.128176 | 0.128351 | 0.127477 | 0.001869 | 0.001579 | 0.003032 |
實驗 2
Weather2k-1var-S4
訓練花費時間:10h 3m 41.84s(註:此實驗非使用單一機器,而是與 Weather2k-1var-S4+AFRK 共用機器進行訓練)
推論階段 AFRK 之 MRTS 基底數:382
| Metric | ALL Locs & All Time | Known Locs & All Time | Unknown Locs & All Time | ALL Locs & Future | Known Locs & Future | Unknown Locs & Future | ALL Locs & Past | Known Locs & Past | Unknown Locs & Past |
|---|---|---|---|---|---|---|---|---|---|
| MSPE | 1.160546e+01 | 8.335466e+00 | 2.465048e+01 | 8.308004e+01 | 8.508390e+01 | 7.508604e+01 | 3.855207e+00 | 1.334625e-02 | 1.918156e+01 |
| RMSPE | 3.406679e+00 | 2.887121e+00 | 4.964925e+00 | 9.114825e+00 | 9.224093e+00 | 8.665220e+00 | 1.963468e+00 | 1.155260e-01 | 4.379676e+00 |
| MSPE% | 1.214263e+10 | 2.615445e+09 | 5.014950e+10 | 2.295537e+10 | 2.672051e+10 | 7.935082e+09 | 1.097017e+10 | 1.642926e+06 | 5.472697e+10 |
| RMSPE% | 1.101936e+05 | 5.114142e+04 | 2.239408e+05 | 1.515103e+05 | 1.634641e+05 | 8.907908e+04 | 1.047386e+05 | 1.281767e+03 | 2.339380e+05 |
| MAPE | 1.034689e+00 | 6.119608e-01 | 2.721080e+00 | 5.516763e+00 | 5.541266e+00 | 5.419014e+00 | 5.486807e-01 | 7.745780e-02 | 2.428533e+00 |
| MAPE% | 3.535854e+08 | 7.255939e+07 | 1.474684e+09 | 4.961357e+08 | 5.373860e+08 | 3.315754e+08 | 3.381282e+08 | 2.215650e+07 | 1.598635e+09 |
MERRA2-1var-S4
訓練花費時間:16h 48m 57.28s
推論階段 AFRK 之 MRTS 基底數:301
| Metric | ALL Locs & All Time | Known Locs & All Time | Unknown Locs & All Time | ALL Locs & Future | Known Locs & Future | Unknown Locs & Future | ALL Locs & Past | Known Locs & Past | Unknown Locs & Past |
|---|---|---|---|---|---|---|---|---|---|
| MSPE | 153.309263 | 153.324779 | 153.247198 | 1564.356985 | 1566.560730 | 1555.542007 | 0.304088 | 0.082327 | 1.191135 |
| RMSPE | 12.381812 | 12.382438 | 12.379305 | 39.551953 | 39.579802 | 39.440360 | 0.551442 | 0.286926 | 1.091391 |
| MSPE% | 0.587130 | 0.587618 | 0.585178 | 5.992495 | 6.004252 | 5.945470 | 0.001006 | 0.000272 | 0.003941 |
| RMSPE% | 0.766244 | 0.766563 | 0.764969 | 2.447957 | 2.450357 | 2.438333 | 0.031721 | 0.016505 | 0.062781 |
| MAPE | 3.651232 | 3.550918 | 4.052489 | 33.980246 | 33.997542 | 33.911060 | 0.362544 | 0.249477 | 0.814813 |
| MAPE% | 0.013523 | 0.013196 | 0.014834 | 0.127042 | 0.127182 | 0.126480 | 0.001214 | 0.000836 | 0.002728 |
實驗 3
Weather2k-S4+AFRK
訓練花費時間:(註:此實驗非使用單一機器,而是與 Weather2k-1var-S4+AFRK 共用機器進行訓練)
推論階段 AFRK 之 MRTS 基底數:
MERRA2-S4+AFRK
訓練花費時間:
推論階段 AFRK 之 MRTS 基底數:
實驗 4
Weather2k-S4
訓練花費時間:4h 31m 17.40s(註:此實驗非使用單一機器,而是與 Weather2k-1var-S4+AFRK 共用機器進行訓練)
推論階段 AFRK 之 MRTS 基底數:178
| Metric | ALL Locs & All Time | Known Locs & All Time | Unknown Locs & All Time | ALL Locs & Future | Known Locs & Future | Unknown Locs & Future | ALL Locs & Past | Known Locs & Past | Unknown Locs & Past |
|---|---|---|---|---|---|---|---|---|---|
| MSPE | 2.729977e+01 | 2.688188e+01 | 2.896683e+01 | 2.397620e+02 | 2.747339e+02 | 1.002486e+02 | 4.261693e+00 | 6.368540e-03 | 2.123748e+01 |
| RMSPE | 5.224918e+00 | 5.184774e+00 | 5.382084e+00 | 1.548425e+01 | 1.657510e+01 | 1.001242e+01 | 2.064387e+00 | 7.980313e-02 | 4.608414e+00 |
| MSPE% | 1.143239e+10 | 2.267235e+09 | 4.799498e+10 | 2.030899e+10 | 2.313886e+10 | 9.019811e+09 | 1.046987e+10 | 4.046914e+06 | 5.222121e+10 |
| RMSPE% | 1.069224e+05 | 4.761549e+04 | 2.190776e+05 | 1.425096e+05 | 1.521146e+05 | 9.497268e+04 | 1.023224e+05 | 2.011694e+03 | 2.285196e+05 |
| MAPE | 1.437579e+00 | 1.047657e+00 | 2.993092e+00 | 9.437545e+00 | 1.014651e+01 | 6.609263e+00 | 5.701122e-01 | 6.103477e-02 | 2.600977e+00 |
| MAPE% | 3.696454e+08 | 7.675813e+07 | 1.538062e+09 | 4.665294e+08 | 4.950431e+08 | 3.527794e+08 | 3.591399e+08 | 3.140192e+07 | 1.666586e+09 |
MERRA2-S4
訓練花費時間:2h 14m 26.41s
推論階段 AFRK 之 MRTS 基底數:111
| Metric | ALL Locs & All Time | Known Locs & All Time | Unknown Locs & All Time | ALL Locs & Future | Known Locs & Future | Unknown Locs & Future | ALL Locs & Past | Known Locs & Past | Unknown Locs & Past |
|---|---|---|---|---|---|---|---|---|---|
| MSPE | 183.060926 | 194.459146 | 137.468045 | 1864.618866 | 1987.368928 | 1373.618619 | 0.723318 | 0.047242 | 3.427622 |
| RMSPE | 13.530001 | 13.944861 | 11.724677 | 43.181233 | 44.579916 | 37.062361 | 0.850481 | 0.217352 | 1.851384 |
| MSPE% | 0.680973 | 0.722399 | 0.515269 | 6.939079 | 7.383059 | 5.163158 | 0.002383 | 0.000159 | 0.011281 |
| RMSPE% | 0.825211 | 0.849941 | 0.717822 | 2.634213 | 2.717179 | 2.272258 | 0.048819 | 0.012604 | 0.106213 |
| MAPE | 3.821658 | 3.698350 | 4.314889 | 35.220395 | 36.220564 | 31.219723 | 0.416976 | 0.171845 | 1.397498 |
| MAPE% | 0.013882 | 0.013486 | 0.015466 | 0.129019 | 0.132507 | 0.115069 | 0.001397 | 0.000580 | 0.004666 |
Bonus
前述實驗皆以半年時長作為實驗時間序列,但似乎下修模型 Residual 層數使得模型學習到的側爭效果成效不彰,尤其以 MERRA-2 實驗尤為糟糕。故於本次實驗間額外做兩個實驗,同樣以 MERRA-2 資料集作為測試資料(因為該資料集較大,同樣時間範圍內容易達到 vRAM 上限,以下實驗訓練過程中 vRAM 皆佔有 30 GB 以上),分別測試在降低輸入 batch_size 的情況下,下列實驗參數的表現:
實驗 5 - MERRA2-1var-S4+AFRK-low-batch
本實驗的訓練集與 MERRA2-1var-S4+AFRK 相同,但降低部分參數,同時提升模型層數到與前次 meeting 接近的參數設定,因 vRAM 關係無法完全一致,具體參數如下:
model.yaml
| |
training.yaml
| |
訓練花費時間:35h 11m 12.33s
推論階段 AFRK 之 MRTS 基底數:301
| Metric | ALL Locs & All Time | Known Locs & All Time | Unknown Locs & All Time | ALL Locs & Future | Known Locs & Future | Unknown Locs & Future | ALL Locs & Past | Known Locs & Past | Unknown Locs & Past |
|---|---|---|---|---|---|---|---|---|---|
| MSPE | 164.784290 | 164.888406 | 164.367824 | 1668.971215 | 1671.274799 | 1659.756880 | 1.679684 | 1.545304 | 2.217204 |
| RMSPE | 12.836833 | 12.840888 | 12.820602 | 40.853044 | 40.881228 | 40.740114 | 1.296026 | 1.243102 | 1.489028 |
| MSPE% | 0.632449 | 0.633265 | 0.629184 | 6.413790 | 6.426202 | 6.364142 | 0.005557 | 0.005116 | 0.007321 |
| RMSPE% | 0.795267 | 0.795780 | 0.793211 | 2.532546 | 2.534995 | 2.522725 | 0.074542 | 0.071523 | 0.085561 |
| MAPE | 4.193186 | 4.136441 | 4.420163 | 34.907394 | 34.918514 | 34.862917 | 0.862729 | 0.798626 | 1.119142 |
| MAPE% | 0.015369 | 0.015185 | 0.016103 | 0.130691 | 0.130813 | 0.130204 | 0.002864 | 0.002647 | 0.003731 |
實驗 6 - MERRA2-1var-S4+AFRK-low-day
本實驗同樣降低部分參數,但提升模型層數到與前次 meeting 更接近的參數設定,且減小訓練集的時間序列至 3 個月,具體參數如下:
model.yaml
| |
training.yaml
| |
訓練花費時間:24h 25m 30.20s
推論階段 AFRK 之 MRTS 基底數:301
| Metric | ALL Locs & All Time | Known Locs & All Time | Unknown Locs & All Time | ALL Locs & Future | Known Locs & Future | Unknown Locs & Future | ALL Locs & Past | Known Locs & Past | Unknown Locs & Past |
|---|---|---|---|---|---|---|---|---|---|
| MSPE | 1.346637 | 1.077605 | 2.422764 | 10.404323 | 10.459679 | 10.182895 | 0.364479 | 0.060272 | 1.581304 |
| RMSPE | 1.160447 | 1.038078 | 1.556523 | 3.225573 | 3.234143 | 3.191065 | 0.603721 | 0.245504 | 1.257499 |
| MSPE% | 0.004831 | 0.003865 | 0.008696 | 0.037348 | 0.037534 | 0.036602 | 0.001305 | 0.000214 | 0.005670 |
| RMSPE% | 0.069507 | 0.062171 | 0.093250 | 0.193256 | 0.193738 | 0.191316 | 0.036130 | 0.014640 | 0.075297 |
| MAPE | 0.517593 | 0.406171 | 0.963280 | 2.524476 | 2.528199 | 2.509585 | 0.299979 | 0.176072 | 0.795608 |
| MAPE% | 0.001852 | 0.001454 | 0.003446 | 0.009074 | 0.009087 | 0.009024 | 0.001069 | 0.000626 | 0.002841 |
結論
目前而言,採實驗 6 之參數設定的 MERRA2-1var-S4+AFRK-low-day 實驗在各項評估指標上表現最佳,尤其在未來時間點的預測表現上,MSPE、RMSPE、MAPE 等指標均有顯著降低,且在過去時間點的預測表現也有不錯的成效。故下次實驗應以此參數設定為基礎,進行更多次的實驗以驗證其穩定性與可靠性。
目前問題為,是否需同步降低 Weather2k 之資料集時間序列,仍需教授不吝指導。
實驗快照
本次實驗共使用 4 臺國網中心機器進行,以下呈現其中 3 臺的實驗快照。
論文暫寫
參考資料
- Zhu X, Xiong Y, Wu M, et al. Weather2K: A Multivariate Spatio-Temporal Benchmark Dataset for Meteorological Forecasting Based on Real-Time Observation Data from Ground Weather Stations[C]//International Conference on Artificial Intelligence and Statistics. PMLR, 2023: 2704-2722.
- Juan Lopez Alcaraz 、 Nils Strodthoff(2022)。Diffusion-based time series imputation and forecasting with structured state space models。Transactions on Machine Learning Research。參考自 https://openreview.net/forum?id=hHiIbk7ApW
- SSSD(2022)。GitHub。參考自 https://github.com/AI4HealthUOL/SSSD



