How to interpret this figure...

Ice water content comparison statistics

Mean ice water content

Panel (a) shows mean ice water content (IWC) versus height for the observations and the model. The blue line shows the observed mean IWC, calculated from radar reflectivity and temperature on the vertical grid of the model. When any rain is measured at the ground, uncorrected attenuation (by rain on the antenna, the melting layer and the rain itself) can render the reflectivity unreliable so IWC is not retrieved and is not included in this mean. Similarly, if reliable liquid water path measurements are not available at the site then for ice overlying stratocumulus, the liquid water attenuation will not have been corrected and so IWC is not retrieved.

The magenta line shows the corresponding mean IWC taken directly from the model. Periods of rain and uncorrected liquid attenuation in the observations have been excluded. The radar can have problems in detecting tenuous ice clouds so for a fairer comparison, the red line shows the mean of the model values after filtering to remove ice clouds estimated to be undetectable by the radar, based on the known radar sensitivity and an estimate of the variation of radar reflectivity on a scale equivalent to the horizontal gridbox size of the model (Hogan and Illingworth 2003, J. Atmos. Sci., 60, 756-767). There is uncertainty in this procedure, as indicated by the red error bars, which show the effect of assuming the radar to be 3 dB more or 3 dB less accurate when performing this filtering. Generally where the red and magenta lines diverge significantly, it is difficult to make a confident comparison with the observations. Fortunately, this tends to occur only above 7-8 km.

As rain in midlatitudes tends to be associated with ice clouds, a considerable fraction of the ice may not be included in this comparison. The grey line shows the model mean over all periods that the radar was operating, including periods of rain and uncorrected attenuation. Comparison with the magenta line indicates what fraction of the total ice mass has been sampled.

(ECMWF only) The ECMWF model treats ice cloud and snow separately, with snow not contributing to cloud fraction or ice water content, and hence not playing a role in radiative transfer. The dashed red line shows filtered model IWC but including the contribution from snow, following Hogan et al. (2001, J. Appl. Meteorol., 40, 513-525). This often produces a better agreement between radar and model in mid-levels.

Frequency of occurrence and amount when present

Panels (c) and (e) show mean IWC split into frequency of occurrence and amount when present. Frequency of occurrence is defined as the fraction of time that IWC on the model grid exceeded 10^-5 g m^-3. It is plotted for the observations and for the model, using the various representations from the model as were used in panel (a). Amount when present depicts the corresponding mean IWC but averaged only over those times when IWC > 10^-5 g m^-3. The combination of frequency of occurrence and amount when present can help to diagnose the source of errors in mean IWC. For example, the model frequency of occurrence may be accurate but amount when present not, indicating that the model carries some ice about the right amount of the time (indicating that the humidity field is reasonable), but has trouble diagnosing the right amount when some is present (indicating that the problem lies with the cloud scheme).

The data files contain these two parameters calculated for a whole range of threshold IWCs between 10^-5 and 1 g m^-3.

Probability density functions

Panels (b), (d) and (f) show the probability density functions (PDFs) of IWC in the height ranges 7-12 km, 3-7 km and 0-3 km, respectively (the data files also contain the 12-18-km PDF but this contains significant data only at tropical sites). The blue bars depict the observed PDF while the red bars depict the model PDF after filtering to remove undetectable ice clouds. Where this filtering has removed significant cloud, the magenta bars are visible, indicating the unmodified model. Note that the leftmost bar (IWCs from 0 to 10^-5 g m^-3) is ten times smaller than the actual value; this is mostly clear-sky events. The grey outline bars indicate the PDFs of the full model distribution, including periods when retrievals were not carried out due to rain and uncorrected attenuation. These usually affect the higher IWC values as those tend to correspond to rain events. Points to note in interpreting these panels:

• Panel (b): 7-12 km - in this range the sensitivity of the radar to tenuous cirrus is often a problem. If the red and magenta bars are different then the blue bars of the observations should be compared to the red bars. If the red and magenta bars are very different then there is likely to be a considerable amount of cloud undetected by the radar, so the process of filtering the model is less certain and one should avoid drawing definitive conclusions about the performance of the model in this height range.
• Panel (d): 3-7 km - in this range the radar tends not to have a problem in detecting the clouds present and the comparison should be reliable.
• Panel (f): 0-3 km - this range is frequently at temperatures warmer than freezing in which case no ice will be present in the model or observations.

Skill scores

The quantities discussed so far evaluate the climatology of the model, but pay no attention to whether clouds were predicted in the right place at the right time, i.e. the quality of the forecast. Panels (g) and (h) depict two skill scores that present a measure of how well the individual ice cloud features were predicted, the Equitable threat score and Yule's Q. Both have the property that a perfect forecast scores 1 while a random forecast scores 0. These two skill scores have been chosen because they are known to be relatively insensitive to the frequency of occurrence of the property being assessed (unlike scores such as hit rate or false alarm rate).

The scores are calculated as follows. Firstly a threshold IWC is chosen. A contingency table is defined, such that A is the number of times that IWC exceeded the threshold in both the model and the observations, B is the number of times that IWC exceeded the threshold in the model but not the observations, C is the number of times that IWC exceeded the threshold in the observations but not the model and D is the number of times that IWC exceeded the threshold in neither the model nor the observations. The scores are defined by:

• Equitable threat score ETS = (A-E)/(A+B+C-E), where E is the number of hits that occurred by chance, given by E=(A+B)*(A+C)/(A+B+C+D). This score is an improvement on the Threat score TS = A/(A+B+C), which does not have the property of a score of 0 corresponding to a random forecast.
• Yule's Q skill score Q = (A*D-B*C)/(A*D+B*C). It is closely related to the odds ratio but conveniently ranges between -1 and 1.