Review of GPTCast

The authors present a nowcasting framework which makes use of principles from large language models (LLMs) to generate an ensemble of radar-based precipitation nowcasts for lead times up to two hours. To this end, the radar rainfall images are split up into patches which the tokenizer learns to encode using a finite, optimal “dictionary” of possible tokens. The whole radar image can thus be compressed using this discretized representation. An optimized Magnitude Weighted Absolute Error (MWAE) loss function is used for training the tokenizer, to give a higher importance to extremes.

In the next step, the sequences of radar patch tokens, similarly to words for LLMs, are used as input to predict the next tokens for each patch location. Not just the temporal sequence but also nearby tokens in space, resulting in a 3D context grid, are used as input to predict the next tokens. The authors claim that the obtained precipitation forecasts are both realistic and reliable in a probabilistic sense. They claim that the model is “fully deterministic” and does not require random inputs (see my later comment about this statement).

The dataset used is the 5-minute ARPAE data from two radars in Central Italy. The data is filtered to include only rainy sequences, and from these sequences some extreme cases are selected for the tokenizer test dataset. The rest is randomly divided into training and validation set. From the tokenizer test dataset, a subset is selected to test the forecaster; I have some reservations about this choice which I will detail below. The proposed methods are shown to improve upon a state-of-the-art non-machine-learning based nowcasting method (LINDA).

The authors present a novel and highly original approach to nowcasting based on LLMs, which could cause rapid progress in nowcasting by piggybacking on the fast-developing field of LLMs. For now, however, a big down side remains the high computational cost of both training and inference. The paper is well-written, and the methodology is supported by overall clear figures, although the description of the loss functions and certain parameter choices could use a more in-depth discussion, and some methodological choices could be improved as described below. I recommend publication if the unclear points below are clarified, and the remaining issues are addressed or the choices clearly motivated in the manuscript. 

General comments and questions

“The model is trained without resorting to randomness” - it’s not clear to me what the authors mean by this claim. Is there no randomness used at all (e.g. no stochasticity in the training process / optimization or the batch selection)? Does it also mean that a given sequence of rainfall images will always deterministically produce the same forecast? Either way, why would this be desirable? Further, they contrast this to other methods which “require random input” - it would be helpful if they are more precise about what they mean exactly by this random input. I suppose it refers to the noise fields used in the training of e.g. diffusion models, but the author’s meaning could be made more explicit.

Some details such as the units of the input data could be clarified more. In that respect, the authors do not mention the error sources affecting the weather radar reflectivity images. Nowcasting systems often use multimodal quantitative precipitation estimates (QPE) (for which radar is of course the main source of information in the areas where it is available, but clutter filtering and rain gauge corrections are often applied). I would also be curious to know how the authors handle the large number of dry patches.

The different terms in the VQGAN loss function could be explained in more detail, for example the LPIPS is not mentioned anywhere in the text. The choice of the different parameters could be better motivated (e.g. why the latent space size of 8; was this based on experience/ literature or were other values tested with worse results?)

If I understand correctly, the output of the model is the distribution for a single token at the center of the spatial window (this seems to be the case based on Fig. 2). The spatial domain can be extended by applying a sliding window approach. Please clarify how the resulting target token distributions are combined and how spatial consistency is obtained. Also, what happens at the edge of the domain?

The authors claim that the dual-stage architecture enables realistic ensemble generation and accurate uncertainty estimation, but it is not clear to me why this would not be the case in a different scenario, for example if the two stages were trained simultaneously – this claim could be further clarified or supported with evidence.

How does the spatial tokenizer deal with local extremes (e.g. a 100-year return level)? How can highly efficient codebook usage (100%) be compatible with such rare extreme values?

The sigmoid function in the MWAE indeed gives more weight to high rain rates, but at the same time the saturation of the sigmoid will make that the factor |sigma(x_i)-sigma(y_i)| will be very small even if x and y represent large differences in very high rain rates. Isn’t this problematic, given that the impact on the ground between a 100- or a 200-year return level event is quite substantial?

The authors discard the non-precipitating series, which represents 71.5% of the data. If the model is used for operational nowcasting, it will also receive dry radar images. How is this dealt with?

The authors apply random rotations in the training phase. How do the authors avoid that the model learns patterns that are unphysical in the sense that the dominant wind direction, orographic enhancement of precipitation etc. will not be learned correctly due to these transformations? Is some context (e.g. topography) provided?

Are the scores with units in table 3 calculated for the reflectivity values in dBZ or for the rain rates? Note that it is easier to interpret RMSE than MSE. It would help to somehow indicate which model score is the best one,  e.g. by underlining it.

Figure 6: Just to be sure, does the box really contain 50% of the points or do the vertical and horizontal sides of the box contain 50% of the points, respectively?

Strictly speaking, there’s a methodological flaw in the selection of the model (either with MAE or MWAE) and the reporting of its performance for forecasting. The authors choose a tokenizer variant based on its performance on the test set, and then go on to evaluate the performance of the resulting nowcasting scheme on a subset of the same test set. The resulting model can very well be the best one, but its score on the FTS (which is a subset of the TTS period) is not representative of the performance for new, truly unseen data. I would like to see the performance of an independent (e.g. more recent) event that was not part of the training / validation / test datasets.

Finally, out of curiosity, I would like to know how hard it would be to retrain the model on a different region. Does everything need to be retrained from scratch, or can one start from a pretrained model (or only the VQGAN for example)? This would significantly reduce the high computational cost associated with these kinds of models. 

Minor  and typographical remarks

·      Please explain all acronyms upon first usage (e.g. GPT in line 2 of the abstract).

·      Something went wrong in the placement of the parentheses of the references. E.g. the first reference “ [...] early warning systemsGöber et al. (2023).” should probably have been “ [...] early warning systems (Göber et al. 2023).

p.1

·       Line 1 of the abstract: “method for ensemble nowcast” -> method for ensemble nowcasting

·      “all variability is learned solely from the data and exposed by model at inference for ensemble generation.” This part is also a bit unclear (and “the” is missing).

p. 2:

·      preserving the precipitation field’s structure -> do you mean “structural characteristics”? Because the mentioned methods don’t necessarily preserve the structure.

·      [...] uncertainty that manifests _itself_ as... /or/ that is manifested as

·      [...] including _the_ medium range weather forecasting domain

p. 4:

·      [...] we find _it_ useful

p. 5:

·      Figure 1: explain the meaning of “stop gradient”

p. 7:

·      Square km -> km^2

·      1km resolution per pixel -> a resolution of 1 km by 1 km

p. 9:

·      We analyze the performances of our model -> … the performance of our models (?)

·      Elsewhere: performances -> performance

p. 13:

·      Tendency to underestimation -> tendency to underestimate

p. 14:

·      Not clear what the authors mean by “The model can be declined […]”