© P. Stauning, Published by EDP Sciences 2018

1 Introduction

The Polar Cap (PC) indices, PCN (North) based on magnetic data from Qaanaaq (Thule) and PCS (South) based on Vostok data, reflect the transpolar convection of plasma and magnetic fields driven by the interaction of the solar wind with the magnetosphere. PC index values are calculated from the basic formula (e.g. Troshichev et al., 2006; Menvielle et al., 2011) by:(1) where ΔF _PROJ is the horizontal polar magnetic variation vector (Δ F  =  F  −  F _QL) measured from a reference quiet level ( F _QL) and projected to a specific optimum direction in space considered to be perpendicular to the transpolar forward (noon to midnight) plasma convection direction. The optimum direction is defined through its angle, φ, with the E-W meridian, while the slope, α, and the intercept, β, are used to calibrate the PC index values against the solar wind merging electric field, E _M. The parameters (φ, α, and β), are tabulated for each minute through the year (e.g. http://pcindex.org/supported-material). They are kept invariant over years. The merging electric field is estimated by using the expression from Kan and Lee (1979):(2) where V _SW is the solar wind velocity, B _T the transverse component (B _T = √(B _Y ² + B _Z ²) of the Interplanetary Magnetic Field (IMF), while θ is the polar angle between the Z-axis of the Geocentric Solar-Magnetospheric (GSM) coordinate system and the transverse IMF component. Here, the OMNIweb service (http://omniweb.gsfc.nasa.gov) is used to provide solar wind data based on contributions from IMP8, ACE, WIND, and GeoTail satellite measurements.

A large number of publications have demonstrated the close correlation between (final) PC indices and further indications of geomagnetic activity such as provided, for instance, by the auroral AE, mid-latitude Kp, and ring current Dst geomagnetic indices (e.g. Menvielle et al., 2011, and references therein). In its real-time version, the PC indices could be used to monitor the transfer of energy from the solar wind to the magnetosphere (e.g. Troshichev et al., 2014) and the related intensities of transpolar convection and cross polar cap potentials (e.g. Troshichev et al., 2000). The PC index development could be used to monitor auroral activity and to forecast substorms (e.g. Janzhura et al., 2007; Troshichev et al., 2014). With adequate integration, the PC indices could be used to predict Kp levels, and they could be used in source functions to predict the development in the Dst ring current index (Stauning et al., 2008; Stauning, 2012; Troshichev et al., 2012).

The PC indices are particularly useful to forecast occurrences of the very intense substorms that may threaten power grid systems. In a study of substorm-associated high-voltage power line disruptions in Sweden, it was shown that such events were almost always preceded by “alert” conditions (PCN > 10 mV/m) through an hour or more (Stauning, 2013c). It has also been demonstrated (Stauning, 2018a) that monitoring PC indices could have given 6 h of early warning (“alert”), of which the last two hours were “red alert” (PCN > 15 mV/m), up to the onset of the strong substorm event on 13 March 1989 that caused the major power outage in Quebec.

It is imperative for using the real-time PC indices in space weather monitoring or forecast services that they are reliable and also robust to possible irregularities like temporary disruptions of the supply of polar geomagnetic data. The reliability could be improved by using multiple data sources (Stauning, 2018a). However, the index derivation procedure must also be reliable and robust. The IAGA-endorsed version of near-real time PC index calculations, as will be shown, are extremely vulnerable to various types of magnetic activity and also to even minor irregularities in the data supply which could result in extended intervals of unavailable or, still worse, misleading index values. Thus, an improved scheme for calculation of real-time PC indices is needed.

The IAGA-endorsed near-real time indices are displayed in a screen image at the web portal http://pcindex.org for a continuously updating 24-h window but not stored permanently. A snapshot of these values can be obtained by downloading the most recent index values up to actual time.

The first reported assessment (Stauning, 2018b) of near-real time PC indices in the IAGA-endorsed version was based on comparing PCS values from downloads on 11 Nov 2014 (with real-time indices) and 25 Oct 2017 (with final indices). The analysis disclosed differences of up to 3.65 mV/m between the PCS indices calculated at around download time and the same series of indices calculated 3 years later from the same basic magnetic data. It should be noted that PC index values above a level of 1.5 ± 0.5 mV/m indicate magnetic storm or substorm conditions according to Troshichev et al. (2014). The present work extends the former analysis (Stauning, 2018b) to further examples but provides also analyses of the IAGA-endorsed methods used to derive PC indices in near-real time. The dominant cause of the large differences between reported real-time and final index values has been disclosed. By calculating anticipated differences, it has been found that the method can be expected to generate unjustified excursions of several mV/m in the near-real time PC index values compared to their final values during many cases of moderate to strong magnetic activity.

Section 2 presents a brief summary of the PC index processing as defined by the software built by the index providers. Section 3 provides an illustrative example based on the download on 30 June 2015 to define near-real time PCN index values and the subsequent download of the same series of index values on 25 October 2017 to define the final index values. In Section 4 the methods are analysed in order to detect the specific cause of differences resulting from the different methods used for calculations of real-time and final PC index values. The results are generalised by calculations throughout 2015 to illustrate the relative frequency and magnitude of anomalies in the real-time PC index values compared to their final values. Section 5 discusses the findings, while Section 6 presents alternative methods to generate real-time index values. Section 7 discusses differences between the IAGA and the DMI versions and is followed by the conclusions. A step-by-step procedure for continuous (e.g. online) generation of real-time PC index values is presented in the appendix.

In this work the term “real-time” value is used for index values calculated at the time of download. The “near-real time” values are those calculated within 2 days preceding download time. The term “prompt” is used for index values (including the “real-time” values) calculated within a couple of months before the download. “Final” or “archival” index values are those downloaded at least 2 years later than their time stamps. Controlled and verified index values are usually termed “definitive” by the data centres. However, the PCN values available since 1975 and the PCS index values available since 1994 are presently termed “provisional” by the International Service of Geomagnetic Indices (ISGI) as reported at their web site, http://isgi.unistra.fr.

2 Processing of magnetic data in the IAGA version

The International Association for Geomagnetism and Aeronomy (IAGA) endorsed by Resolution No. 3 (2013) the PC indices in the formulation suggested by the suppliers, Arctic and Antarctic Research Institute (AARI) in St. Petersburg, Russia, and DTU Space in Lyngby, Denmark. The endorsement was given on basis, among others, of documentation summarized in Appendix A (2014) available at the ftp server at DTU Space. In the documentation, reference is made to the index descriptions in Troshichev et al. (2006) and Troshichev (2011). The derivation of the reference quiet level (QL) used for the endorsed index version refers to Janzhura & Troshichev (2008, 2011) (hereafter J&T2008 and J&T2011, respectively).

Calculations of PC index values according to Equation (1) from the observed horizontal magnetic vector data, F , and using the tabulated parameters (φ, α, β) are performed in the following steps:

1. Derivation of the horizontal QL reference level, F _QL.

2. Calculation of the disturbance vector, Δ F  =  F  −  F _QL.

3. Projection of the disturbance vector to the optimum direction (φ), ΔF _PROJ = (Δ F )_PROJ.

4. Scaling of the projected disturbance vector using (α, β), PC = (ΔF _PROJ − β)/α.

In the former PC index version, the basic principle for the reference quiet level is formulated in Troshichev et al. (2006) where (in Sect. 2.1) it is stated that:

Magnetic deviations δD and δH are calculated from a certain level, “curve of quiet day”, which presents the daily variation, observed at the particular station during extremely quiescent days.

For the derivation of the QDC, F _QDC = (X _QDC, Y _QDC) or (H _QDC, D _QDC), reference is made to the methods described in J&T2008. With unlimited data series available before and after the day in question, an interval of 30 days is considered at a time. For each component, an initial 30-days QDC is now formed by superposition and averaging of quiet 20-min samples from within the 30-days interval. The reference day for this QDC is determined by the relative distribution of quiet samples and is usually located close to the middle of the interval.

Next, the 30-days interval is shifted forward by one day at a time and the calculations repeated. The new QDC may refer to the same or to a different day. With continued forward stepping, a sequence of QDCs is generated. Each initial QDC refers to a specific date. For deriving the final 30-days QDCs for each day of the interval, according to J&T2008, “we use two-dimensional bicubic interpolation with subsequent Savitzky-Golay smoothing [Ramsay, 2005]”.

The new feature in the IAGA-endorsed procedure compared to the preceding PC index version defined in Troshichev et al. (2006) is the introduction of a solar wind sector (SWS) vector term, F _SWS = (X _SWS, Y _SWS) or (H _SWS, D _SWS), defined in J&T2011 (see also Menvielle et al., 2011). The solar sector terms are meant to compensate for the effects on the transpolar plasma convection patterns from the Y-component, IMF B _Y, of the interplanetary magnetic field coupled to the solar wind sector structure. The changing convection patterns generate systematic variations in the polar magnetic fields particularly at daytime (e.g. Stauning, 2015). The steady SWS-related variations could be considered part of the quiet level rather than part of disturbances in PC index calculations.

According to J&T2011, and also specifically mentioned in the documentation in Appendix A (2014) submitted to IAGA for the endorsement process, and stated in messages from the index providers, the SWS terms are only used for correcting the measured magnetic data from which the QDCs are calculated. The index reference level would then be F _QL, _SWS =  F _QDC, _SWS.

However, Appendix A (2014) also holds transcripts of essential computer programs used for calculation of PC index values. They enable tracking of the general data processing scheme, while the more subtle details are hidden in machine coded subroutines. In this software, the solar sector terms appear in the derivation of the QDC from data corrected for the SWS values as well as directly in the definition of the reference level, that is, F _QL,_SWS =  F _SWS +  F _QDC,_SWS, from which the disturbance components are measured. The expression in Equation (1) for the PC index could be written: (3)

The direct inclusion of the median-based solar wind sector term, F _SWS, into the reference level from which the disturbances involved in the calculation of PC index values are measured, generates an effective QL level that may not (most likely does not) represent the quiet level found at corresponding local time, seasonal, and IMF B _Y conditions (Stauning, 2013a, 2015). This is a violation of the basic principle for the reference quiet level formulated in Troshichev et al. (2006).

The QL problems related to the SWS-terms in posterior data processing are aggravated in near-real time processing. The solar wind sector terms are derived from forward extrapolation of the cubic spline functions fitted to the four 3-day median values derived for every other of the preceding 9 days. The precise definition is provided in J&T2011 and quoted in Section 4 here.

For the extension of the QDC procedure to real-time applications, where only past data are available, the most recently completed 30-days QDC, according to J&T2008, is projected forward in time to the actual day by using the seasonal trend derived from last year’s QDCs. However, there is now a potential problem since, contrary to the cases dealt with in J&T2008, where no SWS effects are involved, previous QDCs are now derived from SWS-corrected values and not straight from the raw data. The SWS-related effects on previous QDCs, whether for the same or from the preceding year, may not repeat at the actual interval in question. Hence, the SWS effects on the real-time QDCs could differ substantially from the SWS effects acting on the final QDCs for the same dates. The problem is not dealt with in the available documentation.

The real-time cubic spline construction of the SWS-terms is prone to generate more variable QL reference levels which, particularly at night time conditions, may deviate substantially from the quiet levels found during corresponding local time, seasonal and IMF B _Y conditions. The different methods used to derive the SWS terms and also the general differences in the methods and the data bases used for the involved 30-days QDCs generate differences between values of PC index series calculated in near-real time and values of the same index series calculated posterior from the same basic magnetic data (see Stauning, 2018b). The present work analyses and discusses such differences more closely.

3 Example case

The near-real time PC index values are not permanently available. However, comparisons of PC index values downloaded from the (otherwise excellent) web server http://pcindex.org at different times may disclose features of the derivation procedures that are not illustrated in the available reference publications or in other documentation provided for the IAGA endorsement. Such downloads (unfortunately) have been made just occasionally. However, the comparisons between the available cases of near-real time PC index values calculated at around the time of download and the final values made available several years later might provide useful indications of essential problems in the index calculation methods. An example is presented in Figure 1.

Fig. 1. Comparison of PCN values for June 2015 from download from http://pcindex.org on 30 June termed “prompt” values and download on 25 October 2017 termed “final” PCN values. The upper panel presents (in blue line) values of the solar wind merging electric field (E M) and (in red line) the IMF B Y component (OMNI data). The bottom panel presents the differences between prompt and final PCN values.

One download which included the near-real time values at that time took place on 30 June 2015, 15:57 (time stamp on the WinZip file). The PCN index values termed “prompt” from this download are presented in second panel from the top of Figure 1 holding one month’s data. The download of the PC index values termed “final” took place on 25 October 2017, 11:28. The PCN data from this download are presented in the third panel from the top. The differences between the prompt and final PCN values are presented (without any smoothing) in the bottom panel. For assessments of the solar wind and IMF conditions, the top field displays hourly values of the merging electric field, E _M, (blue line) and the IMF B _Y component (red line). These values were derived from OMNI solar wind bow shock nose (BSN) data comprising contributions from IMP8, ACE, WIND and Geotail measurements made available from the OMNIweb service at http://omniweb.gsfc.nasa.gov.

Throughout the exposed interval from 1 to 30 June 2015 the magnetic data are the same for the two downloads and the calibration parameters (φ, α, β) are also identical. Consequently, as seen from the expression in Equation (3), the only cause of possible differences between the PCN values from the two downloads resides in the different QL reference levels, F _QL. This is also evident from the smoothness of the differences which is in strong contrast to the variability seen in the values of the individual index series. The sinusoidal-like variations seen in the differences are mainly the results of the projection to the optimum direction fixed with respect to the solar wind-magnetosphere system of the difference vector Δ F _SWS =  F _SWS(near-real time) – F _SWS(final) since both SWS vectors are derived on a daily basis in coordinates fixed at the rotating Earth.

A prominent feature seen in Figure 1 is the large differences between the near-real time (prompt) and the final PCN values during the days close to the date of downloading (30 June) of the former series. A further feature to note is the continued irregular differences between prompt and final PCN values throughout the month proceding the download time. Note also in Figure 1 the discontinuities seen at day shifts (at 00:00 UT) in the patterns of the PCN differences. From corresponding displays (not shown) for April and May of prompt data from the download on 30 June 2015, and final data from the download on 25 October 2017, the irregular differences as well as the discontinuities are seen to continue back to 2 April 2015.

On the download day (30 June 2015) the differences range between −1.4 and +1.6 mV/m, while on 23 June the differences peak at +3.09 mV/m. Thus, the present case adds another example of excessive deviations between near-real time and final PC index values to the case reported in Stauning (2018b). The available cases of prompt values from occasional downloads in 2014 and 2015 are presented in Table 1. The download of final values was made on 25 October 2017. Extreme values of the differences are emphasized in the table. The downloaded PC index data are available in the Supplement.

4 Analyses of differences between real-time and final PC index values

It seems that the near-real time PC index values provided at http://pcindex.org are not kept for further assessments of their validity. Thus, representative values are available from occasional downloads only (e.g. Table 1). The present work, therefore, tries to resolve the cause of these differences attempting to derive more general indications of their possible frequencies and magnitudes and their relations to other conditions. Unfortunately, many details of the derivation of the SWS terms and the QDCs are not available in the reference publications (J&T2008 and J&T2011) or in the documentation in Appendix A (2014). Direct communications with the index and software providers have also been fruitless so far. Thus, such details must be inferred from examination of the available index values, e.g., in displays like Figure 1.

As mentioned above, the differences between near-real time and final index values must be related to differences in QL values since the magnetic data and the calibration constants are the same. The irregular patterns of the differences during the 3 months from 2 April up to the date of the download of the prompt PCN values indicate that the differences should be found mainly in the SWS terms since the 30-days QDC would hardly change that much over a few days (cf. examples of smooth QDC variations in J&T2008). The discontinuities at transitions between days indicate that the SWS-terms are shifted each day.

In summary, it is inferred that the main part of the differences between real-time and final index values on the download day arise from shifts between using forward extrapolation of the cubic spline constructions based on four 3-day medians derived every other of the foregoing 9 days like detailed in J&T2011 and the subsequent smoothing of medians at the processing much later. For the differences between prompt and final values observed on other than the download day, no explanation of the applied calculation method is provided in the available documentation.

Assuming that the 30-days QDCs are about the same in the near-real time and final versions, the differences between PC indices in the two versions could be calculated from the differences between the SWS-vectors using appropriate projection and calibration parameters. The derivation of the real-time solar sector terms is detailed in J&T2011 pp. 1496–1497 by the following procedure:

1. median values for magnetic components H and D are derived for four intervals of days preceding with the exception of the current day (n = 0):

   • r1 = F[for interval from n-3 day to n-1 day]

   • r2 = F[for interval from n-5 day to n-3 day]

   • r3 = F[for interval from n-7 day to n-5 day]

   • r4 = F[for interval from n-9 day to n-7 day]

2. piecewise polynomial form of the cubic spline interpolant for r1, r2, r3, and r4 segments is determined;

3. termination of this form related to day n = 0 is examined as representative of the SWS effect for the current day, even if this day is disturbed.

It might be noted that the illustration in Figure 7 of J&T2011 of the results from using the above procedure is seriously misleading. In contrast to the description in the text, the displayed H _SWS values are the post-event smoothed median values for the final version and not the real-time cubic spline extrapolated H _SWS values, which are much more variable (see Stauning 2018b).

Figure 2 displays results derived using magnetic data from Qaanaaq (Thule) throughout June 2015. The E _M and IMF B _Y values displayed in the top field of Figure 2 are the same as those shown in Figure 1. The middle fields of Figure 2 display by the dots connected by a green line the 3-day median values of the X and Y components derived as the average of 3 daily medians. The cubic spline extrapolated 3-day medians (real-time solar sector terms) are displayed by dots connected by a red line while the smoothed median values (final solar sector terms) are displayed in magenta line.

Fig. 2. Calculations of anticipated differences between real-time and final PCN indices. E M (blue line) and IMF B Y (red) in top field (OMNI data). Three-day medians (green) of X and Y components, extrapolated 3-day medians (red) and smoothed median values (magenta) are displayed in the middle fields. Differences between real-time and final PCN values are shown in the bottom field for each day separately.

In Figure 2, an illustrative example shows the X-component cubic spline construction (in black line) based on 3-day medians from 17, 19, 21 and 23 June (marked by black dots) extrapolated to provide the real-time X _SWS value for 25 June (marked by a large black dot). A corresponding example for the Y-component shows the construction (in black line) of the real-time Y _SWS value for 25 June (large black dot). The example case for X _SWS on 25 June is shown in greater detail in Figure 3.

Fig. 3. Details of Fig. 2. Cubic spline construction (black line) of the solar wind sector term, X SWS, for 25 June based on 3-day median values from 23 (r1), 21 (r2), 19 (r3) and 17 June (r4). The 3-day median values are displayed in green line, real-time X SWS in red line. Final (smoothed) X SWS in magenta line.)

In Figure 3, the green lines connect 3-day median values. The four 3-day median values named r1–r4 (cf. J&T2011) used for the cubic spline construction have been marked by black dots superimposed on the green ones. The cubic spline function is displayed in black line including the tangential extension up to the X _SWS value on 25 June marked by a large black dot. The 3-day median values on 24 and 25 June connected by a green dashed line were not available on the date, 25 June, for the cubic spline construction shown here. Note the “take-off” of the cubic spline tangential extension beyond the 3-day median value (green) on 23 June (the r1 point) to define an excessively high value for X _SWS on 25 June compared to the 3-day median value (green) for the day or the smoothed median value (magenta) that defines the final X _SWS value on that day.

The red lines connect previously determined real-time X _SWS values, while the curve in magenta line displays the smoothed solar wind sector X-component terms derived in the post-event processing. The difference (here 100 nT) between the real-time and the post-event X _SWS values in combination with the corresponding difference between the real-time and the post-event Y _SWS values and with projection and calibration processing by Equations (4)–(6) form the large differences ranging from −2.0 to +4.4 mV/m between anticipated near-real time and final PC index values on this day, 25 June 2015.

The solar wind sector components, X _SWS and Y _SWS, from Figure 2 provide the solar wind sector term F _SWS in the real-time as well as in the final versions for each day of June. To derive index differences, these vectors (in their rotating coordinate system) are projected to the optimum direction using the optimum angle values (φ) provided at http://pcindex.org for each minute of the involved days. The projection angle is: (4)where λ is the longitude of the observatory (THL), while hrs is UT time in decimal hours of the day.

The projection is accomplished through Equation (5):(5)

The SWS-related differences between the real-time and final PCN values can now be calculated according to Equation (3) by the differences between the projected SWS vectors divided by the slope coefficients (α) provided at http://pcindex.org:(6)

The varying values of ΔPCN are displayed in the bottom field of Figure 2 for each day of June 2015 separately. Note, that these values represent the anticipated differences between near-real time and final values to be found on the day by watching the display or downloading near-real time PCN index values (e.g. for space weather monitoring) from http://pcindex.org. On 30 June 2015 the calculated differences reach values between −1.8 and +2.8 mV/m which should be compared to the differences between the downloaded near-real time and final values presented in Figure 1 that range between −1.4 and +1.6 mV/m. Complete agreement could not be expected since the 30-days QDCs may differ between the real-time and the final versions, but the order of magnitude agrees.

Depending on the day in June 2015 of simulated download of real-time PCN index values, the differences between real-time and final values may range between −4.07 (28 June) and +4.43 mV/m (25 June). Results from corresponding calculations for further months of 2015 gave the representative anticipated monthly peak differences listed in Table 2. The numbers of days where the anticipated numerical values of either ΔPCN or ΔPCS exceeded 1 resp. 2 mV/m are also listed for each month. The extreme values in each column are emphasized.

5 Discussions

The most prominent features of Table 1 are the large differences between real-time and final PCN indices ranging from −1.79 to +3.09 mV/m and the still larger differences between real-time and final PCS indices ranging from −2.21 to +3.65 mV/m. The quoted differences can easily be verified from the PC index files downloaded from http://pcindex.org at different times and made available in the Supplement. It should be noted that the differences are not resulting from poor data quality. They are, predominantly, the adverse results from the method used to derive the solar wind sector terms in real time through the applied cubic spline construction. For judging these values, PC index values above 1.5 ± 0.5 mV/m are considered by Troshichev et al. (2014) to indicate magnetic storm or substorm conditions.

Figure 4 provides a more detailed view of the differences than available from displays like Figure 1. The figure displays in red line the near-real time indices from the actual download (2015b) on 30 June 2015 listed in Table 1. The final indices from the much later download on 25 Oct 2017 are displayed in magenta line throughout 2 days including the real-time download day. The shifts in the polarity of the differences are related to the projection of the solar wind sector difference vectors, (X _SWS(prompt) − X _SWS(final), Y _SWS(prompt) − Y _SWS(final)), defined in rotating Earth coordinates to the optimum direction fixed in space.

Fig. 4. Display of PCN values from download of near-real time index values (red line) on 30 June 2015 (File 2015b from Table 1) and final index values (magenta line) from download 25 Oct 2017.

Note in Figure 4 the characteristics of the differences up to the time of download (at 15:57 Danish summer time = 13:57 UT on 30 June 2015). Between 08 and 14 UT on 30 June the large negative values of the real-time PCN index values indicate strong reverse convection conditions while the final PCN indices for the same interval show index values close to zero indicating quiet conditions. Thus, the two index versions indicate quite different geophysical situations.

In Figure 5 from a corresponding download (2014c) of near-real time index values, the large peak in the real-time PCS index values at around 06 to 09 UT on 11 Nov would indicate the onset of a strong geomagnetic storm. During real-time space weather monitoring, the rapid rise towards an index level of 5 mV/m would be particularly alarming. In contrast, the final index starts out at around 06 UT on 11 Nov from large negative values indicating reverse convection conditions and turns into fluctuating small positive and negative values indicating rather quiet conditions. Thus, the two index versions indicate substantially different geophysical conditions.

Fig. 5. Display of PCS values from download of near-real time index values (red line) on 11 Nov 2014 (Case 2014c from Table 1) and final index values (magenta line) from download 25 Oct 2017.

Examining further intervals of the displays in Figures 4 and 5 shows several corresponding cases of very different indications of the geophysical conditions from the near-real time and the final PC index series, respectively.

For an examination of the cause of the large differences, an inspection of Figure 2 discloses that the large differences between the forward projected and the smoothed SWS terms, and thus the excessive differences between prompt and final PC index values, occur at or just after intervals where the IMF B _Y components display considerable variability. The variations in the IMF B _Y values generate corresponding variations in the median values of the magnetic components. In the cubic spline construction, the forward projection extending 2 days ahead of the latest 3-day median value amplifies such variations (described in a non-professional term as the “roller-coaster take-off” effect) and could even give changes to a different polarity of the real-time SWS terms compared to the final median values.

In the X _SWS example for 25 June 2015, shown in Figures 2 and 3, the forward projection extending from the 3-day median on 23 June shows strongly increasing values while the actual 3-day medians as well as the final (smoothed) X _SWS values on 24 and 25 June show decreasing values. The corresponding feature is seen in the Y _SWS example from 25 June and in many further cases. This mechanism is considered the main cause of the excessive differences between near-real time and final PC index values in the IAGA-endorsed versions.

Thus, the cubic spline-based method to determine the SWS terms in real time aggravates the QL problem residing in the posterior version (cf. Stauning, 2015) since the SWS terms may now fluctuate strongly from one day to the next (cf. Fig. 2) instead of the more damped variations in the smoothed solar wind sector terms. It should be noted that the calculations of the solar wind sector terms from the recorded magnetic data using the prescribed methods (J&T2011) are accurate and “final”. The differences between the forward projected real-time and the final settled 30-days QDCs appear unpredictable; at best they are fairly small. The differences between the real-time and the final index values resulting from differences in the QDCs could either mitigate or come on top of the differences resulting from the different derivation of the contributions from the solar wind sector terms.

It is seen from Table 2 that the anticipated difference throughout 2015 between real-time and final PCN index values range between −4.07 and +4.43 mV/m. Comparing these values with the corresponding differences ranging from −2.21 to +3.65 mV/m shown in Table 1 derived from the few available downloads of near-real time indices indicates agreement in order of magnitude. The agreement substantiates the relevance of the calculations of anticipated differences. Thus, Table 2 could also be considered to indicate the general frequency and order of magnitude of expected differences between real-time and final PCN (or PCS) index values.

Table 2 shows that the differences between real-time and final indices are generally strongest and most frequent in the PCN index values compared to PCS indices. Furthermore, in both PCN and PCS index versions, the differences are strongest and most frequent during the local summer season. In May or June, 2015, the anticipated PCN differences exceeded 1 mV/m every other day and ranged above 2 mV/m during every fourth day on the average.

Values from July 2015 are not included in the series of summer months presented in Table 2. In the otherwise very regular supply of magnetic data from Qaanaaq (THL), the recordings from 13–17 and 19–24 July are missing or incomplete. Reliable real-time PCN indices in the IAGA-endorsed version could not be derived in an extended interval between 14 July and 2 August due to the complicated construction of the cubic spline interpolation polynomials for the real-time SWS values. Thus, the method used to generate real-time solar sector terms makes the derived PC index values highly vulnerable to temporary disruptions of the data supply.

Even worse, incomplete data, e.g. missing data through some hours of a day, may change the daily median values by considerable amounts. Such anomalies are amplified by the cubic spline construction and may migrate to generate incorrect near-real time PC index values throughout the 9 days where the damaged median value is involved among the four 3-day medians.

In order to judge the effects of incomplete magnetic data, the calculations leading to estimates of the differences between near-real time and final PCN index values were repeated in a special version. The magnetic data from the first half of 23 June, 2015, were made unavailable. Thus, the medians for this day had to be calculated from the remaining data from 12:00 to 24:00 UT only. The results are displayed in Figure 6b. The initial results (also presented in Fig. 2) are displayed in Figure 6a for comparison.

Fig. 6. (a) Continuous data available. Top panel: X-component 3-day medians (green), smoothed (final) medians (magenta), SWS terms (cubic spline extrapolants) in red line. Middle: Y-component medians and SWS terms. Bottom: differences between near-real time and final PCN index values. (b) Same format and terms with magnetic data from 00:00 to 12:00 UT on 23 June made unavailable.

It is clear from the differences between Figures 6a and b that the absence of magnetic data from the first 12 h of 23 June generates substantial changes. The 3-day median values (green) are changed throughout 21–25 June where they include the median value for 23 June. The SWS cubic spline extrapolated values (red line) and thus the near-real time PCN values are changed throughout 22 June to 3 July, while the 7-day smoothed medians (magenta) and thus the final PCN values are changed throughout 20–26 June.

PCN values, of course, are missing throughout the interval of missing data, i.e., from 00:00 to 12:00 UT on 23 June. The maximum changes in the final PCN values are around 1 mV/m on 23–24 June. The largest changes in the near-real time PCN index values are around 4 mV/m and occur on 25 June, that is, 2 days after the 12 h interval of missing data imposed on 23 June. The differences between the final and the near-real time PCN index values range up to 3 mV/m on 25 June as seen in the difference between the negative peak values of −2.9 in Figure 6a and −5.9 mV/m in Figure 6b.

6 Alternative real-time PC index derivation methods

In order to provide constructive suggestions for improvements of the derivation procedure, alternative methods for calculations of the reference quiet level (QL) are outlined. Basis for the suggested alternative real-time QL derivation scheme is the “solar rotation weighted” (SRW) QDC method developed at the Danish Meteorological Institute (DMI) and presented in Stauning (2011).

The SRW method is part of the “DMI” PC index derivation procedure which has been used to calculate indices presented in publications since 2007 (see references in Stauning, 2016). The scaling parameters, (φ, α, β), in the DMI version are almost the same as those of the IAGA-endorsed version during winter days and summer night times but differ markedly during summer daytime conditions. The solar wind and geomagnetic data bases (1997–2009) as well as the correlation and regression methods used for calculations of index parameters are the same for the two versions, but the DMI procedure omits strong reverse convection samples. Illustrations of the associated effects on index quality are provided, e.g., in Stauning (2013b, 2015, 2018a).

With the SRW method, the final QL is the QDC constructed by weighted averaging of quiet samples for corresponding times of the day from an interval of ±40 days from the day in question. In addition to giving weights according to the “quietness” of the recorded values, the samples are weighted to give preference to dates closest to (and including) the day in question and to dates with nearly the same view of the Sun in its 27 days rotation.

In real-time calculations of PC indices, the QL estimates use only past data recorded 40 days prior to the date and time in question. The use of pre-event data only generates differences between the real-time and the final QDCs. These differences are largest at equinoxes where the seasonal variations in the QDCs are largest and the post-event data, therefore, are most needed to balance the anticipated seasonal QDC changes. In post-event calculations the QL estimates are gradually improved as samples from up to 40 days following the day in question become available.

The robustness of the QL methods in the DMI real-time and final versions to irregularities in the basic supply of magnetic data was tested the same way as the corresponding test of the IAGA-endorsed versions by making data from 00:00 to 12:00 UT on 23 June unavailable for the QL calculations. The reference QLs for the real-time as well as the final versions were changed by insignificant amounts, less than 1 nT, corresponding to changes in the PC indices of less than 0.03 mV/m. The reason for the small changes in the DMI versions of the QLs and the large changes in the IAGA QLs is the same. The high disturbance level on 23 June 2015 generates minimal values to the quietness weights for the samples used for the SRW method, while the SWS term in the IAGA QL depends strongly on the interval of data used for deriving the daily median value.

The appendix holds an outline of a step by step procedure for deriving real-time, prompt, and final QL and PC index values in the DMI versions from observed magnetic data assumed to be currently available. A programmer, most likely, could easily convert the procedure into an operational program in his programming language. If the magnetic data are not changed later on then the derived PC indices are turned into final values after 40 days. During intervals of missing data, the PC indices, of course, could not be derived. However, if the interruption lasts less than a few days, then reliable PC index calculations could be resumed at once data become available.

It might be noted that magnetic data from observatories within the central northern polar cap like Qaanaaq in Greenland, Alert, and Resolute Bay in Canada would give about the same positive PCN index values in the DMI version using coefficients derived without including reverse convection samples. Negative index values that would indicate reverse convection could be quite different. Usually, however, reverse convection conditions relate to weak global magnetic activity. Correspondingly, magnetic data from observatories in the central southern polar cap like Vostok and Dome-C, would give about the same positive PCS index values (Stauning, 2018a).

7 Comparison between IAGA-endorsed and DMI versions

Differences between near-real time and final PC index values have been calculated separately for the IAGA-endorsed versions and the versions developed at DMI (Stauning, 2016). The results for the summer months, May to July, 2015, are shown in Figure 7.

Fig. 7. Hourly values of E M (blue line) and B Y (red) from OMNI solar wind data are shown in top field. Differences between PCN index values calculated by using real-time and final values of F SWS in the IAGA-endorsed QL versions in the middle field and the corresponding differences for the DMI versions in the bottom field. Intervals of missing values are marked.

The most important factor for the different relations between the near-real time and the final PC index values in DMI versions vs. the IAGA-endorsed versions is the different QL derivation scheme. For the PC indices in the DMI versions the QL values are in both cases estimated from the quietest samples throughout an extended interval.

In contrast, the IAGA-endorsed methods to derive the reference QL include solar wind sector terms derived from median values that could show large variations in periods with magnetic disturbances. Such variations are reduced by smoothing the SWS terms in the final version but amplified (at times strongly enhanced) by the cubic spline extrapolation process thus creating large variations in the derived near-real time PC indices and large differences with respect to the final index values. Differences between 30-days real-time and final QDCs may add further fluctuations to the displayed difference values for the IAGA versions.

The peak differences between real-time and final indices based on hourly values are noted in the respective fields of Figure 7. For the DMI version, the maximum deviations range between −0.11 and 0.41 mV/m throughout the summer season. Throughout the year (2015) the maximum differences (not shown) range between −0.74 and +0.45 mV/m, which could be considered “tolerable” in view of the uncertainty range of ±0.5 mV/m associated with the 1.5 mV/m onset level for magnetic storm activity found by Troshichev et al. (2014). The ranges for the DMI versions should be contrasted with differences between the near-real time and final indices ranging from −1.79 to +3.09 mV/m observed at occasional downloads (Table 1) of PCN data in the IAGA versions or with the anticipated range from −4.07 to +4.43 mV/m (Fig. 2 or Table 2) derived throughout 2015 by using the prescribed IAGA-endorsed calculation methods for the solar wind sector terms included in the QLs.

In addition to reducing the differences between real-time and final PC index values by almost an order of magnitude compared to the IAGA-endorsed methods, the SRW method developed at DMI to derive the QL levels is far more robust to data irregularities. Like shown by the robustness test (ΔPC < 0.03 mV/m) and also demonstrated in Figure 7, intervals of some hours or a few days void of data would only have minor effects on the resulting QDCs. PC index values from such intervals, of course, could not be restored, but reliable index values could be obtained throughout all neighbouring intervals.

For the IAGA-endorsed versions, the complicated construction of the SWS terms implies that the real-time as well as the final SWS terms could not be derived for lengthy intervals extending beyond temporary disruptions in the supply of data. Such intervals are marked in the middle field of Figure 7. Still worse, the derived PC indices are possibly given incorrect values at irregularities in the supply of magnetic data. The test example displayed in Figures 6a and b showed that missing data in 12 h of a day could change the difference between near-real time and final PC index values throughout 14 days by up to 3 mV/m. On top, the near-real time index would change by up to 4 mV/m on 25 June at 2 days after the 12 h of unavailable data on 23 June.

Conclusions

• In spite of the adoption of IAGA Resolution No. 3 (2013) where “IAGA … recommends use of the PC index by the international scientific community in its near-real time and definitive forms”, and availability of the index series from http://pcindex.org since February 2014, no assessment of the relations between the near-real time and archival (final) PC index values has been reported until the recent analysis provided in Stauning (2018b). This analysis was based on occasional downloads of index values since the near-real time indices are not permanently available.

• The present work reports on observed differences of up to 3.09 mV/m for the PCN indices and 3.65 mV/m for the PCS indices derived from comparisons of real-time index values in 2014 and 2015 derived from occasional downloads from http://pcindex.org and later downloads from the same source in 2017 of the same index series in their final version. These data are available in the Supplement.

• By using the published IAGA-endorsed calculation scheme, a systematic assessment of differences between real-time and final PC index values throughout 2015 has disclosed anticipated differences of up to 4.43 mV/m. Cases with differences above 2 mV/m are expected to have occurred during more than 20 days of that year.

• Noting that PC index values above a level of 1.5 ± 0.5 mV/m are considered to indicate magnetic storm or substorm conditions (Troshichev et al., 2014), the large observed or anticipated differences between real-time and final PC index values imply that real-time indices may report magnetically quiet conditions while the same indices in their final version would report strong magnetic storm (or vice versa).

• The cubic spline extrapolation method involved in the near-real time IAGA-endorsed index version uses four 3-day median values of the magnetic data to derive the solar wind sector terms used for the QL reference level from which disturbances are measured. This construction makes the derivation of PC index values highly vulnerable to conditions with varying IMF B _Y levels. At such conditions the daily median values of the recorded magnetic data may fluctuate strongly. Such fluctuations are further amplified by the cubic spline extrapolation process to create excessively high (or low) reference levels.

• In addition, the method needs 9 days in succession with good data for each cubic spline construction and, therefore, is strongly vulnerable to irregularities in the supply of magnetic data. In a test example where data were made unavailable for 12 h, the PC index values were changed throughout 14 days. The near-real time indices were changed by up to 4 mV/m at 2 days after the interval of unavailable data. Thus, a 12 h disruption of the flow of magnetic data could make PC indices indicate (or hide) strong magnetic storm activity 2 days later regardless of actual conditions.

• Alternative methods to provide more reliable real-time QL levels and PC index values are outlined in the article. A calculation scheme which could be used, for instance, for space weather monitoring by users with access to polar magnetic data either from the standard observatories (Qaanaaq and Vostok) or alternative polar cap stations, is specified in the appendix.

• IAGA and the index providers have been notified of the frequent occurrence of large differences between near-real time PC indices obtained from http://pcindex.org and the same index series in the final version. The alternative methods described here might be considered an option for possible modifications of the present IAGA-endorsed PC index derivation methodology.

Data availability

Near-real time PC index values and archival PCN and PCS index series derived by the IAGA-endorsed procedure are available through the web site: http://pcindex.org. The (otherwise excellent) web site holds a display of current 24 h near-real time PCN and PCS values and provides also a download option for the index values. The near-real time PC index values are not stored and, therefore, not permanently available. The web site, furthermore, holds PCN and PCS index calibration coefficients. Solar wind sector terms and QDC values are not included. The web site includes the document “Polar Cap (PC) Index” (Troshichev, 2011). Files of near-real time and final PCN and PCS data used in the present paper are provided in the Supplementary Material and available at https://www.swscjournal.org/10.1051/swsc/2018031/olm.

Geomagnetic data from Qaanaaq and Vostok supplied from the INTERMAGNET data service center were downloaded from http://intermagnet.org.

Solar wind OMNI BSN data from combined ACE, WIND, IMP8, and Geotail interplanetary satellite measurements were provided from the OMNIweb data service at Goddard Space Flight Center, NASA, at http://omniweb.gsfc.nasa.gov

The “DMI” PC index versions including the real time and final QDC calculation methods are documented in the report SR-16-22 (Stauning, 2016) available at the DMI web site: http://www.dmi.dk/fileadmin/user_upload/Rapporter/TR/2016/SR-16-22-PCindex.pdf.

Supplementary Material

Files of near-real time and final PCN and PCS data Access here

Acknowledgments

The staffs at the observatories in Qaanaaq and Vostok and their supporting institutes, DMI, DTU Space, and AARI, are gratefully acknowledged for providing high-quality geomagnetic data for this study. The comprehensive service at the OMNIweb data center (http://omniweb.gsfc.nasa.gov) to provide processed solar wind satellite data, the efficient provision of geomagnetic data from the INTERMAGNET data center (http://intermagnet.org), and the excellent performance of the PC index portal (http://pcindex.org) are greatly appreciated. The author gratefully acknowledges the collaboration and many rewarding discussions in the past with Drs. O.A. Troshichev and A.S. Janzhura at the Arctic and Antarctic Research Institute in St. Petersburg, Russia.

The editor thanks two anonymous referees for their assistance in evaluating this paper.

References

Appendix

Calculation of real-time and final PC indices by alternative (DMI) methods

The appendix outlines a step-by-step procedure for calculation of real-time (and final) Polar Cap (PC) indices from a continuously available source of polar geomagnetic data. The magnetic data are assumed to be provided in a 1-min cadence either from direct connection to a polar cap magnetometer or from on-line internet access to the data.

PC index calibration parameters for the selected observatory are assumed available. See Stauning (2016) for calibration parameters for Thule and Vostok. Calibration parameters for Resolute Bay and Dome C observatories are available on request (cf. Stauning, 2018a).

The main challenge is the determination of the quiet level (QL) from which the disturbances involved in calculation of PC index values are measured.

Basically, the QL level for any given day is formed by weighted averaging of hourly samples of the magnetic data recorded at corresponding hours throughout an interval of ±40 days (-40 days to present at real time calculations). The samples are weighted to give preference to the quietest samples, recorded closest to the day in question, and having the same view to the rotating Sun. The average values are derived from the sum of weighted samples divided by the sum of weights.

The QL and PC index calculations could easily be handled by a typical personal computer.

Initialisation

• Observatory polar cap polarity (N or S) and longitude (λ) are defined.

• PC index calibration parameters, angle (φ), slope (α), and intercept (β) are loaded, e.g., in their monthly average hourly values version (Stauning, 2016). They are invariant over years. The values are 2-D interpolated and stored, e.g., to provide hourly values throughout the year taking the leap/normal year succession of days into account (or interpolated into minute values if storage capacity permits).

• Calculation of tables of weight function values (see Stauning, 2011 for details):

Daily weight function for separation: ; XDD = DD − D0 is the separation in days from day, DD, to the actual day, D0. RDD is date difference normalization constant (=40 days).

Daily weight function for solar rotation: ; LSR (=27 days) is solar rotation period.

These two weight function can be combined: WF_DR = WF_SR·WF_DD. Weight function values are derived for XDD = 0–40 days and stored.

Hourly weight function for smoothing: is the separation in hours. RHR is hour difference normalization constant (=6 h). Weight function values are derived for XHR = 0 to 2 RHR (=12) hours and stored.

Actions each minute

• Horizontal magnetic components (X _OBS, Y _OBS) are acquired and stored (missing values replaced by error code to restore a continuous series).

• Calibration parameters (φ, α, β) interpolated to the actual minute (e.g., from their hourly values)

• QL components (X _QL, Y _QL) interpolated to the actual minute (e.g., from their hourly values).

• Magnetic variations measured from QL are calculated: ΔX = X _OBS – X _QL; ΔY = Y _OBS – Y _QL

• Projection angle (V _PROJ) calculated from: V _PROJ = λ + 15° hrs + φ (hrs is time of day in decimal hours)

• Projection of horizontal magnetic variation vector to optimum direction: ΔF _PROJ = ΔX sin(V _PROJ) ± ΔY cos(V _PROJ) (+ for southern, − for northern polar cap)

• Calculation of real-time PC index: PC = (ΔF _PROJ − β)/α

• Real-time PC index value displayed and stored.

Actions at the start of each hour

• Calculation of the average values, X _M(H0 − 1), Y _M (H0 − 1) of the previous hour’s 1-minute magnetic component values (error code samples omitted). H0 is the present hour of the present day, D0.

• Calculation of data variability weight values for the previous hour. Two variance parameters are calculated. One is the maximum time derivative of the 1-min samples of the horizontal field vector F through the hour, i.e. the maximum change, δFt, in vector magnitude from one minute to the next:

This parameter provides an indication of the “smoothness” of the values recorded through the hour.

The other parameter is the average variance in the horizontal field vector F through the sample interval, i.e.:where δX(t) = X(t) − X _M and δY(t) = Y(t) − Y _M are the deviations from the mean values, (X _M ,Y _M). This parameter provides an indication of the slope of the values recorded through the hour.

For the sample interval (1 h) with the calculated variances a weight function related to the “normalized” variances is defined as shown below:

The X and Y component average values (X _M, Y _M) and their assigned variability weight function value, WF_VV (same for both components) are stored. Note that the hourly average component values and their variability weight function values are not changed later on, while the date difference and solar rotation weight function values are recalculated at each new calculation of QLs.

Actions at the start of a new day

• Calculation of predicted hourly QL values for all hours of the present day (D0) and also for the first hour of the next day (for interpolation through the last hour of the day). First, initial QL values are calculated for an interval extended by ±12 h in order to accomplish smoothing of the values.

For each horizontal component (X, Y) and for all hours (HX = 0–23) of the present day (DX = D0), the last 12 h of the preceding day (DX = D0 − 1, HX = 12–23), and the first 13 h of the next day (DX = D0 + 1, HX = 00–12), initial QL values are calculated by weighted superposition of stored hourly average samples and associated variability weights for corresponding hours through the previous 40 days, DD = D0–40 to D0–1.

The initial QL values for each hour, H0, of the day, D0, and the first hour of the next day, D0 + 1, are then smoothed by weighted averaging through H0 − 12 to H0 + 12 h. XHR = −12 to +12 h is the separation in hours:

Missing (error code) samples are omitted in the summations.

Corresponding formulas apply to the Y component, QLY. The vector-based weight function values are the same for the X and Y components.

The calculated hourly QL values are stored and thus ready for interpolation to any minute of the present day for PC index calculations.

Display of PC index values

A succession of stored real-time PC index values is now available for display, e.g., for monitoring of index values through the past 24 h. It is recommended for added reliability when used for space weather monitoring that corresponding PC index values are calculated from alternative sources of magnetic data whenever possible (Stauning, 2018a).

For best possible quality of extended displays or listings, the past PC indices could be recalculated by using the above calculation scheme. For each day in the past, the summation to determine the initial QL values is extended over the previous 40 days, the selected day, and further on up to the present day. The smoothing scheme is the same. When the selected day is 40 days or more away from the present day, then the calculated PC indices are “final” (but still “provisional”).

Finalization of PC index values

When, typically after a couple of years, the magnetic data are “definitive” (see INTERMAGNET data service at http://intermagnet.org), then the PC indices could also be made definitive. The PC index calculations would still follow the above outlined scheme. However, the interpolation technique should be improved recalling that the hourly averages of interpolated values should restore the original hourly values (this condition excludes simple linear, parabolic, and cubic spline interpolation methods). Furthermore, when deriving the weight functions for the weighted summations over ±40 days, the decimal values of the days should be used in order to include the hour of the day in the calculation of separation values. Correspondingly, decimal hour values should be used to include the minute of the hour in the calculation of weights included in the smoothing scheme.

The resulting PC index values should be visually inspected in order to exclude obviously corrupted samples and, whenever possible, compared to index values derived using alternative sources for the polar magnetic data (see Stauning, 2018a). It would also be useful to compare PCN and PCS indices. Due to the common calibration with respect to the solar wind merging electric field, the two index series, in principle, should provide the same values regardless of differences in latitude, local time and season.

The solar wind merging electric field values involved in the calibration procedures use the square of the IMF B _Y component value. Hence the B _Y polarity is irrelevant for this purpose. However, the convection patterns in the northern and southern polar caps are differently affected by the IMF B _Y conditions which may give different PCN and PCS values. The effects of systematic variations in the IMF B _Y values with the solar wind sector structure are mitigated by using the SRW QL procedure.

All Tables

All Figures

Fig. 1. Comparison of PCN values for June 2015 from download from http://pcindex.org on 30 June termed “prompt” values and download on 25 October 2017 termed “final” PCN values. The upper panel presents (in blue line) values of the solar wind merging electric field (E M) and (in red line) the IMF B Y component (OMNI data). The bottom panel presents the differences between prompt and final PCN values.

In the text

Fig. 2. Calculations of anticipated differences between real-time and final PCN indices. E M (blue line) and IMF B Y (red) in top field (OMNI data). Three-day medians (green) of X and Y components, extrapolated 3-day medians (red) and smoothed median values (magenta) are displayed in the middle fields. Differences between real-time and final PCN values are shown in the bottom field for each day separately.

In the text

	Fig. 3. Details of Fig. 2. Cubic spline construction (black line) of the solar wind sector term, X SWS, for 25 June based on 3-day median values from 23 (r1), 21 (r2), 19 (r3) and 17 June (r4). The 3-day median values are displayed in green line, real-time X SWS in red line. Final (smoothed) X SWS in magenta line.)
In the text

	Fig. 4. Display of PCN values from download of near-real time index values (red line) on 30 June 2015 (File 2015b from Table 1) and final index values (magenta line) from download 25 Oct 2017.
In the text

	Fig. 5. Display of PCS values from download of near-real time index values (red line) on 11 Nov 2014 (Case 2014c from Table 1) and final index values (magenta line) from download 25 Oct 2017.
In the text

	Fig. 6. (a) Continuous data available. Top panel: X-component 3-day medians (green), smoothed (final) medians (magenta), SWS terms (cubic spline extrapolants) in red line. Middle: Y-component medians and SWS terms. Bottom: differences between near-real time and final PCN index values. (b) Same format and terms with magnetic data from 00:00 to 12:00 UT on 23 June made unavailable.
In the text

	Fig. 7. Hourly values of E M (blue line) and B Y (red) from OMNI solar wind data are shown in top field. Differences between PCN index values calculated by using real-time and final values of F SWS in the IAGA-endorsed QL versions in the middle field and the corresponding differences for the DMI versions in the bottom field. Intervals of missing values are marked.
In the text