Headlines > News > SpaceX Update: Findings of the Falcon Return to Flight Board

SpaceX Update: Findings of the Falcon Return to Flight Board

Published by Sigurd De Keyser on Wed Jul 26, 2006 6:28 pm
More share options

Below is a summary of the findings of the board, convened by DARPA and with participation by SpaceX, NASA, the US Air Force and the US Naval Research Laboratory. The Co-Chairs of the board were Brigadier General (retired) Pete Worden, who is now the NASA Center Director at Ames, and Colonel (retired) Bob Paulson, who has 30 years experience in space-related program management, systems engineering and research and development with the Air Force.

Video, photo and data confirm that an engine fire started just after ignition, but prior to liftoff. The fire continued to burn during all powered flight. The telemetry shows that at:

  • T-400s, when the propellant pre-valves were opened, a fuel leak started on the turbopump fuel inlet pressure transducer plumbing. Data shows that this same leak was not present on the test firing that occurred on March 22.
  • ~T-2s, ignition of the engine in turn ignited the fuel leak, initiating the engine fire.
  • ~T+25s, the engine fire burned through a pneumatic line, causing pneumatic pressure to drop.
  • T+30s, the fuel and oxygen pre-valves began to close as pneumatic pressure dropped below minimum levels.
  • T+34s, engine thrust terminated due to closure of propellant pre-valves.

Fortunately, telemetry analysis allowed us to pinpoint the leak to a very specific location on the vehicle, clearly showing that the leak arose at the turbopump fuel inlet pressure transducer. This is supported by several cameras and video feeds around the launch site, as well as launch vehicle parts recovered from the ocean.
While there is a high degree of certainty that a fuel leak fire is what ended the flight and the location of the leak is clear, there is some uncertainty as to how the leak arose. In all the SpaceX engine tests (numbering in the hundreds), vehicle wet dress rehearsals and launch pad static firings, there has never been a fuel leak of any significance. In what appears to be a stroke of extremely bad luck, it happened for the first time on launch day.

After reviewing several possible, but unlikely, causes of the leak, the board concluded that the most plausible scenario was that the aluminum B-nut on the transducer failed shortly before launch due to stress corrosion cracking. Early in the investigation, it appeared as though a pad processing error might have caused the leak, but recovered hardware showed the lockwire that led to the B-nut was attached to the fuel line. If the stress corrosion cracking explanation is correct, it means that, over the course of almost four years, the first and only time we saw an aluminum B-nut crack was on launch day. “Unfortunate” would be an understatement.

Ironically, this is a case where a more expensive part failed! We make use of the more expensive and lighter aluminum B-nuts, rather than the cheaper and heavier stainless steel B-nuts. Frankly, and this may surprise some people, but there is not a single case I’m aware of on the Falcon 1 or Falcon 9 where we chose the cheaper and less reliable component over the more expensive and higher reliability component. Those that would ascribe this failure to reach orbit to cheap parts are incorrect.

The B-nuts are anodized for corrosion protection, but the ones mounted on the exterior of the engine panel did see considerable exposure to the environment, which might have overcome the anodized coating. The maiden launch campaign stretched over several months and had numerous countdowns where the vehicle was exposed to the elements for days or weeks at a time. It is worth noting that anything not exposed to the elements, such as the interior of the main engine pyramid, interstage, avionics bay or inside of the payload fairing, showed no evidence of corrosion.

The above represents the determination of the Return to Flight Board as to why the rocket shut down prematurely. Now, I will delve into the improvements. These are a superset of those recommended by the board, so should be seen as my words, not necessarily an official opinion of the board.

Improvements to the Falcon Launch System

The Falcon launch system in place for our upcoming launch will be better in several ways. I say “launch system”, because the improvements apply to the rocket itself, the software that monitors vehicle health and the people driven processes for executing the launch. I will address each of the three categories, excluding only proprietary and ITAR restricted information:

1. Vehicle Design

None of the changes are fundamental and, to the best of our knowledge, none are an absolute necessity prior to launch. For example, the corrosion issue could be addressed by simply limiting the amount of time that the vehicle sits on the launch pad or putting a protective canvas cover over the engine compartment until launch. However, the changes do improve vehicle robustness and further the SpaceX goal of achieving a “Maytag” rocket, where it just doesn’t break.

All changes are being analyzed and delta qualified. The government review team is also reviewing the improvements, of which the list below is a subset:

  • Exposed aluminum B-nuts have been replaced with either an orbital welded joint, as used by high end satellites, or a stainless steel B-nut where a weld joint is not possible. In addition to addressing the stress corrosion concern, stainless steel also removes the potential for galvanic corrosion, since the lines are stainless steel.
  • A one way (check) valve has been placed at the fuel leak check port. When combined with a safety wired cap, this provides dual redundancy for sealing the leak check port. This technique is also used on high end satellites.
  • The fuel vent has been directed further from the engine bay and into the airflow free stream, ensuring that fuel vapors cannot feed a fire.
  • Engine bay openings will be covered by fireproof blankets and the engine bay will be continuously purged by nitrogen. Even if there is another fire, this should prevent damage to engine electronics, control valves and much of the pneumatic system.
  • Teflon lined flex hose, which was used in the pneumatic system, has been replaced with a corrugated metal flex line with a high temp overwrap. This will allow the pneumatic lines to survive a fire should one ever arise again.
  • The robustness of the avionics system wiring has been improved to reduce servicing needs at Kwajalein. In defense of the avionics, I should point out that, despite being immersed in fire, the avionics system operated correctly throughout the entire flight!
  • The first and second stage now transmit telemetry independently. This results in better grounding and hence cleaner, less noisy data, which is critical for detecting problems early. Also, we can continue receiving first stage data after stage separation.
  • The foam insulation will be bonded on, rather than removed by lanyards. The foam’s late separation appeared to have no effect on the vehicle operation, but there is potential for problems to arise with the lanyard system. At the expense of a slight mass increase, bonding on the insulation, as is done with the upper stage, was felt to be the most robust solution.
  • The launch mount arms will rotate further out of the way to improve engine clearance during high ground winds.

2. People & Processes

Considerably more detail has been added to the written procedures for doing work on the rocket at the launch pad. New procedures to deal with unexpected anomalies will require approval of the VP of Launch Operations and the VPs of any department affected by the activity.

Any work done on the rocket, whether a routine part of launch preparations or resolving an anomaly, will now require three signoffs, including the technician, responsible engineer and quality assurance. Previously, work had required only the technician and responsible engineer to cross-check.

This is actually not a significant cost increment, as previously the technician and engineer were responsible for completing quality assurance paperwork. By having the QA person take care of logging all the information and entering it into our QA database, it alleviates the workload of the technician and engineer and allows them to stay focused on preparing the rocket for launch.

In addition, the QA person will take digital close out photos of all work done and these will be examined by the launch readiness review team prior to giving the green light.

3. Software Health Monitoring, Launch Automation and Ground Support Equipment (GSE) Upgrades

would characterize the design improvements to the rocket as modest, essentially a version 1.1 of the original. However, the improvements to the software that monitors the vehicle health prior to launch are considerably more significant, arguably version 2+.

More than 700 parameters are now tracked continuously, including both the vehicle and critical ground support equipment. This is an order of magnitude greater than in the past, where only parameters known to be potential problems were tracked. The new software tracks literally everything, even sensors that are not critical to launch, but whose misbehavior could tangentially indicate a real problem.

Values are examined both instantaneously and as trends that might show an issue developing. When the system detects a potential problem, a warning message is displayed on the launch console. If the problem is determined to be critical, the software will halt the countdown and set the rocket and GSE to a safe state.

From being a long series of manually executed steps, the launch countdown is now mostly automated, reducing the potential for human error. Engineers are still on station watching the rocket telemetry, but, much like the autopilot helps alleviate pilot workload in an airliner, they have more time available to analyze the vehicle state. This improvement allows for reduction of the launch control room crew (those executing rote tasks) without reducing the amount of attention paid to the rocket.

For our next launch, we have configured a control room at our California headquarters with a real-time telemetry relay and direct access to the launch control voice networks. This is essentially a remote extension of the Kwajalein control room, albeit with read only access, and allows us to double the number of engineers on console. However, because they are only needed in the hour prior to launch, their time is very efficiently used. No need for the expense of travel, lodging and time spent waiting for launch to happen at Kwajalein (another efficiency improvement).

There are a number of other improvements to the ground support equipment, only some of which I will mention here:

  • The clean room for satellite processing has been expanded and upgraded for processing of larger and more complex spacecraft.
  • The vehicle hangar is now double-walled for insulation, allowing precise control of temperature and humidity.
  • Payload air conditioning while in the fairing has also been upgraded to allow for wide control over temperature and humidity.
No comments
Start the ball rolling by posting a comment on this article!
Leave a reply
You must be logged in to post a comment.
© 2018 The International Space Fellowship, developed by Gabitasoft Interactive. All Rights Reserved.  Privacy Policy | Terms of Use