Outage at the Stanford, JSOC is Offline

A cooling water line burst in the Stanford building housing the SDO HMI JSOC yesterday. Several inches of water were in the JSOC room. As of this morning it appears the flooding has been drained but things are still wet and at least one leak onto electronic equipment continued. An initial assessment is that the water affected many systems but did not provide a time for return to service.

The JSOC is currently offline while the damage is assessed and repaired. The SDO data will be stored at the DDS in New Mexico until the repairs are complete and then will be automatically delivered to the JSOC.

This means that all HMI and AIA data from the JSOC will be unavailable until repairs are completed, the JSOC restarted, and the data delivery from the DDS re-established. Science data will be produced several days after that. Near-realtime data from before 26 Nov 2024 will continue to be available on the SDO wenbsite.

We appreciate your understanding, and are grateful for the dedication and responsiveness of the Stanford team. 

Network Outage Causes Data Problems

A cable cut in New Mexico is causing some of the SDO telemetry to be delayed to the science teams. As a result, the teams do not know the roll angle of SDO and the images appear to be rotating as we execute the HMI Roll Calibration today between 1500 and 2140 UTC (11:00 am - 5:40 pm ET). The rotation angle will be provided in due course and the science data will have the information to be correctly oriented with solar North up. Once the maneuver is finished the images will again be correctly oriented.

JSOC Power has been Restored

Power has been restored to Stanford University and all of the components of the JSOC have been turned on. Near-realtime data has been flowing for a day. The science data pipeline will beginning serving data as soon as possible.

Power Still Out at Stanford

The power is still out at the Stanford University building that houses the AIA/HMI JSOC. AIA and HMI data will be unavailable until power is restored. This outage is apparently due to a wildfire near Palo Alo, CA, that brought down the main power lines into the area. The downlinked data is being stored at SDO's White Sands ground station. Once power is restored that data will be transmitted to the JSOC and made available for study.

EVE data is still available at the EVE SOC.

Snow Disrupts Data

Accummulated snow in the SDO-1 antenna caused an outage in the SDO Ka science data from approximately 1815 UTC to 2310 UTC (1:15 pm ET to 6:10 pm ET) on February 4, 2022. The weather is improving in Las Cruces and we anticipate no further outages.

A Week of Infrastructure Repairs!

This week is a busy week for people making repairs of SDO-related infrastructure.

On Tuesday, 8 June 2021, the fiber optics linking the SDO ground station and the SDO JSOC in Palo Alto, CA, will be disconnected and relocated. This work is scheduled to start at noon MT and to take about 2 hours but could last as long as 6 hours. During this time the AIA and HMI data will be unavailable. The automatic replay option will fill in the data once the repairs are complete.

The NASA building housing the SDO servers will undergo infrastructure maintenance 12-13 June 2021. This will require us to shutdown our servers over that weekend, which means the website and data will not be available. The servers will be powered down Friday evening (June 11, 5 pm ET) and powered up Monday morning (June 14).

We apologize in advance for any inconvenience these repairs may cause.

Power Interruption Today

The SDO.gsfc.nasa.gov website will be unavailable today from 7:00 am - noon ET while work on the electrical supply to the building takes place.

Short Outages in the SDO Website and Data Flow

Since Monday there have been several interruptions in the SDO website and data flow. On Monday a test of the Inertial Reference Units resulted in SDO pointing slightly off center for about 90 minutes. Some data may not be created during that time. On Monday and Tuesday our system administrators were updating the SDO website and, perhaps in anticipation of their impending replacement, one of the older RAIDs failed. There have been several interruptions as the SAs repair the issues and these will continue this week. SDO data continues to flow to the data centers.

Data Outages Have Ended

The data outages caused by snow and ice ended Monday morning when the Sun melted the last pieces of ice off of the SDO antennas.
Many thanks to the FOT members who worked to keep the antennas pointed at SDO through a holiday weekend.

Data Outages Due to Snow in Southern New Mexico

Since last night, the SDO science telemetry downlink has been severely degraded due to snow and ice in southern New Mexico. The telemetry files being delivered to the SOCs have been partial for all three instruments. Accumulations of ice and or snow have probably collected on both SDO1 and SDO2 dishes. At the moment, conditions are still poor. The data outages will stop when the severe weather ceases.

Data Delivery System in Maintenance Mode

The SDO Data Delivery System (DDS) is the interface between the antennas that receive the SDO data and the science data centers. At this time we are performing maintenance on the DDS and hope to have it back in service soon. Science data is being cached and will be delivered to the SOCs after the system is restarted.

Yesterday's Maneuver and Today's Weather in Las Cruces

Yesterday at 1950 UTC (2:50 p.m. ET) SDO did Momentum Maneuver #21. Science data may not be available for 20 minutes on either side of the thruster firing.

There is heavy snow at the White Sands Complex near Las Cruces, NM. About 6" of snow has fallen so far. Snow can cause the Ka-band science downlink to fade, and it completely faded at 1815 UTC (1:15 p.m. ET) and has not yet returned at 1915 UTC (2;15 p.m. ET). The S-band link is not affected by the snow and is still being received. When the snowfall abates we will again receive the science data.

Update 23-Jan-2015: Normal science data flow was restarted at 2204 UTC yesterday (5:04 p.m. ET). Many thanks to the people who cleaned the snow off the antennas!