A Primer on Aircraft Test & Evaluation Principles
Lieutenant Colonel L. D. Alford
This paper is a primer for both aviators and non-aviators on how test pilots conduct flight test. It describes the basic types of flight test, and for each type of test answers the following five questions:
- What is the aeronautical concept being tested?
- How are analysis, modeling, simulation, and ground tests carried out and used?
- How is the flight test accomplished?
- What is done with the data?
- How does the concept affect an aircraft?
To be considered safe for flight in the United States, a civil aircraft type must be certified by the Federal Aviation Administration (FAA) in accordance with the Federal Aviation Regulations (FARS). The process of certification begins with the aircraft components and ends with certification flight test. Although, the FAA is the keeper of the certification process, the aircraft and aircraft component manufacturers are the driving force behind the test of the aircraft. They build, design, and test aircraft systems not only to meet the requirements of certification, but to manufacture outstanding aviation products that meet their customers’ needs. T&E is the engineering tool that proves an aircraft’s capabilities and ultimately demonstrates certification, but certification is only a small part of T&E. In the development of aircraft and aircraft systems, the rubber meets the road in Test & Evaluation (T&E). The T&E process starts with the first drawing on the back of an envelope, it follows through the design of each component and confirms flight safety and utility, and it helps design, develop, and build the aircraft and all of its systems. T&E starts with analysis and ends with analysis. It is an experimental science that works hand in hand with engineering to produce a safe and affordable aircraft. The manufacturer has a greater concern than certification; the manufacturer wants to design an aircraft that is easy and fun to fly. In this article, I will explain how T&E helps achieve this goal.
Why do we accomplish flight test?
Flight test is a key part of the engineering and aircraft certification process. It accomplishes three objectives: airworthiness, mission suitability, and certification. Airworthiness is the flight-proven envelope and performance characterization of an atmospheric flight vehicle, and it is the most important objective of the three because it determines the limits and peculiarities of the aircraft. Airworthiness is the bottom line. If airworthiness is not demonstrated and its limits defined, the resulting aircraft may not be safe for the average aviator to operate. The second objective, mission suitability, is the proven ability of a vehicle to perform a defined mission. For example, a Cessna 152 is a workhorse training aircraft, but it would flunk as an acrobatic or heavy cargo aircraft. Likewise, a Lear 35 would not make the best initial flight training aircraft, but it is a great executive transport. Not only must an aircraft be airworthy, but to be usable, it must also be suitable for the mission. If either airworthiness of mission suitability is lacking, the aircraft is unacceptable. The last objective of flight test is certification compliance. Although, an aircraft may be both airworthy and mission suitable, it may, at first, not meet all certification requirements. This is a typical outcome for many aircraft, and in this case, the manufacturer works along with the FAA to modify or redesign the aircraft to meet the certification criteria.
Basic Scientific Principles in Flight Test
Flight test is an experimental science that uses the scientific method as the foundation of its approach. To apply the scientific method, we first develop a theory and then design an experiment (a test) to prove that theory. Next, we run the test and, finally, we analyze the resulting data to determine whether the theory was correct. This is the approach used in flight test.
The development of the theory for any specific flight test is accomplished through analysis, modeling, simulation, and ground tests. Analysis is the paper or computer study of a flight test problem and produces a result that can be measured. For example, a hand analysis of the estimated takeoff distance for a new aircraft will result in a number—the estimated takeoff distance. This theoretical number can then be checked using a flight test experiment. We can also make use of a computer to analyze the takeoff. This inquiry will also result in a discrete takeoff distance for the aircraft. To get a better theoretical number, we can model the aircraft in a computer or a wind tunnel and simulate the takeoff. This modeling and simulation (M&S) will also result in a theoretical number for the takeoff distance. Finally, we use ground tests to both help check the theoretical numbers and to approximate the actual takeoff distance. Before we accomplish any flight test we usually try to determine at least two theoretical numbers to predict the results of the test. Two numbers allows a crosscheck and gives a line for statistical analysis. For dangerous tests, such as takeoff testing, ground tests always precede flight tests. A test plan describes the analysis and defines how the test will be safely accomplished. For a takeoff test, the test plan will contain a matrix (table 1) that includes an analysis and an M&S result for the takeoff distance. This matrix will require ground test and flight test to fill in the blanks. If the results of the ground or the flight test are significantly different from the analysis and M&S, the analysis and M&S must be reviewed and corrected so current and future numbers agree within statistical significance (usually 3 s ). Statistical significance and confidence intervals are the bread and butter of flight test analysis. In general, flight test for safety of flight must be accomplished to show a 99% statistical confidence while non-safety of flight tests must show a 95% statistical confidence. Statistical confidence is a measure of surety and simply means that we are 99 or 95% confident in the results. In other words, there is a 1 or 5% chance respectively, that we are wrong.
|Hand Analysis||Computer Analysis||Ground Test||Flight Test|
|Takeoff Test #1|
|Takeoff Test #2|
Table 1: Takeoff Test Matrix Example
The last two concepts I want to describe in this section are test discipline and build-up. Test discipline is the consistent application of the test process to prevent mishaps and to ensure good test data. Test discipline means that the events the test crew flies are carefully defined and orchestrated well before the mission starts. Without strong test discipline, the data may not be acceptable and the flight test program can risk an aircraft mishap. Build-up is the term used to describe a gradual approach that starts at a safely defined point in the aircraft flight envelope and then increases critical parameters such as airspeed and g-loading to define the edges of the envelope. When it refers to decreasing parameters, this process is sometimes called a build-down. Build-up (down) is the only logical and safe approach to most flight test.
In the next sections, I will describe each major area of flight testing and tell you:
- What aeronautical concept is being tested.
- How analysis, modeling, simulation, and ground tests are carried out and used.
- How the flight test is accomplished.
- What is done with the data.
- How the concept affects an aircraft.
All of the testing and phenomenon described in each of these areas encompass and support the three objectives of test and evaluation–airworthiness, mission suitability, and certification.
Electromagnetic Interference (EMI) and Electromagnetic Compatibility (EMC)
Electronic components, instruments, and controls are a significant technological trend in modern aircraft. Electrical systems produce electromagnetic fields around wiring and can output interfering signals into other electronic components. The fly-by-wire flight controls, control-by-wire engine controls, and other aircraft systems (e.g., radar, electrical production, radios, instruments, etc.) all have the potential to interfere with one another. In addition, electronic systems can also interfere with materials (i.e., detonate, ignite, damage), communications, navigation, and other electronic ground and ship systems inside and outside the aircraft. Any of these effects can result in aircraft mishaps and cause systems degradations. A good example of this is the interference of AM radio frequencies with the original Tornado’s fly-by-wire flight control system. This problem caused the loss of more than one aircraft before the manufacturer corrected it. I have seen interference during ground test for EMI/EMC and in lab testing, but the worst EMI I experienced was when portable electronic equipment in the passenger compartment caused unexpected music over aircraft intercom systems and the VORTAC needles dance to the music.
Analysis, M&S, lab testing, and ground testing are used extensively to predict and uncover potential EMI/EMC problems. Analysis and M&S investigate the theoretical interactions between systems. In lab testing, the component or system is bombarded at high power with a range of possible interfering frequencies. This type of testing discovers most interference problems and is essential for aircraft components; however, because lab testing does not emulate all aircraft electrical and hardware systems interactions, it cannot discover all possible interference. Ground testing is necessary to fill in the blanks from the lab and see how the component acts in the full aircraft environment. During ground testing, unlike the lab, we check only selected discrete signals and problem areas determined in the lab and from analysis. If no problems were found in the lab, the device is only tested at normal power levels and representative high, medium, and low frequencies. Ground testing is accomplished by a method termed source/victim. In source/victim, the article to be tested is turned on and subsequent aircraft systems are operated one by one. Any discrepancies in either the existing or the new aircraft systems are recorded and analyzed. Next, the process is reversed and the new system is operated while the existing aircraft systems are examined for proper operation. Because of the complexity of aircraft systems and the digital nature of modern components, ground testing alone rarely discovers all EMI/EMC interactions.
Source/victim EMI/EMC ground testing is part of the basic airworthiness testing of the aircraft, and is usually one of the final hurdles before the aircraft is allowed to begin flight test. Most flight testing for EMI/EMC is accomplished as an ad hoc part of the test program. In the air, during the evaluation of the aircraft systems, particularly the instrument systems, the test aviators watch for interference and other EMI/EMC problems. EMI/EMC is so critical to flight safety almost no in-air testing is accomplished with known problems.
EMI/EMC testing is used to identify and correct interference problems. It can also be used to limit aircraft. For instance, because some radio transmissions can cause the instruments to display incorrectly, most aircraft are restricted from the use of certain HF radio frequencies during instrument (ILS) approaches. In addition, EMI/EMC is the reason the Federal Aviation Administration (FAA) does not allow portable electronic equipment to be turned on in airliners below 10,000 feet.
EMI/EMC problems will have a significant effect on an aircraft. In general, these problems must be corrected before the flight test can continue and the cost in dollars, schedule, and technical risk can be high. For example, interference problems on both the C-130 electrical upgrade and the Directed Infrared Countermeasures (DIRCM) led to significant weight growth due to added wiring and component shielding. However, not all EMI/EMC problems require costly fixes, as the above ILS-HF interference example shows, sometimes limits can be placed on the use of equipment to reduce safety problems.
If you experience EMI/EMC on your aircraft, ensure all non-FAA-approved electronics are turned off. With some equipment you may have to remove the batteries. If that doesn’t work, turn off all portable electronics. It is highly unlikely but not impossible for approved aircraft equipment and modifications to cause EMI/EMC. If you discover an unexpected EMI/EMC problem on your aircraft not attributable to portable equipment, immediately notify the manufacturer/modifier and have the aircraft inspected.
Currently, a pitot-static system is the most common method we have of measuring airspeed and altitude. To display airspeed, the airspeed indicator measures the force of the air, dynamic pressure (q), through the aircraft’s pitot tube and compares it to static pressure from the static ports. Dynamic pressure is greatest with increasing airspeed, acceleration, and increasing air density. The difference between dynamic and static pressure is converted to a measure of speed. The altimeter and vertical velocity indicator use static pressure to give, respectfully, height above a pressure datum (usually adjusted to show approximate height above sea level) and climb/descent rate. The Mach meter measures temperature along with dynamic and static pressure to give the speed of the aircraft relative to the speed of sound. Because the airflow around an aircraft varies with altitude, airspeed, and attitude, the uncorrected readings from pitot-static instruments can display wildly inaccurate information. Electronic and mechanical computers are used to apply corrections to the raw data and to stabilize readings. So the pilot receives accurate information throughout the flight envelope, all pitot-static systems must be calibrated to determine these corrections.
Much effort goes into the computer analysis and M&S of the pitot-static system. Unfortunately, although extensive work is accomplished to support this testing, it rarely predicts the results well. The MC-130H, AC-130U, and Advanced Range Instrumentation Aircraft (ARIA) KC-135 are all aircraft whose pitot-static systems required significantly greater testing than planned. Specifically on all three of these aircraft, only enough time and money was budgeted for a basic look at the system, but all three aircraft required multiple complete recalibrations. Though exact pitot-static analysis is difficult, this kind of study and M&S is critical to the aircraft development because the data allows the design of the pitot-static system and provides a basis for flight evaluation. Ground testing is of limited utility, but it is also important. Most airplane airspeed systems don’t work until approximately 25-50 Knots indicated airspeed (KIAS) and the systems are always in “ground effect” during ground testing. “Ground effect” is the interaction between the ground and the aircraft and affects the aircraft from ground level to up to one half the wingspan above the ground. Because of this interaction, the aircraft requires less energy to fly, and the airflow around the aircraft is different than in flight away from the ground. In spite of “ground effect” interactions, during high-speed taxi tests (ground tests) enough data can be gathered to check the system and support a first flight takeoff.
Flight test of pitot-static systems can be accomplished using four different methods: tower fly-by, pace/chase, ground/air measurement, and speed course. Tower fly-by is the most accurate and common method used to calibrate a pitot-static system. In this flight test technique (FTT), the test pilot (TP) flies at a designated, indicated airspeed, ground-track, and altitude out of “ground effect” (usually 100 feet) beside a ground based measurement facility. In the facility, the flight test engineer (FTE) measures the actual aircraft altitude. The altitude is compared with the aircraft’s indications to determine instrument corrections. The second method, pace/chase uses a calibrated chase aircraft that flies in formation with the test aircraft. Both pilots take steady data to compare the test aircraft systems with the calibrated aircraft. The accuracy of this method is limited to the chase aircraft’s calibration, but this approach is generally required to check high altitude points and airspeeds greater than Mach one. The third method, ground or air measurement is accomplished through GPS, laser, or radar tracking, and uses aircraft position data compared to indicated altitude and airspeed to determine pitot-static corrections. These methods are usually less accurate than the others, but they are increasing in acceptance and accuracy. The fourth and final method is speed course. Although it is the least accurate method, it has some advantages. Specifically, it doesn’t require any special equipment and can be completed from the aircraft without outside support. In conducting a speed course test, the TP flies a measured ground track at a set altitude (usually 100 feet). The airspeed, calculated from the time to travel over the ground track, is used to calibrate the other instruments. Speed course is often combined with ground or air measurement techniques to improve the data.
Because almost every system and flight test requires accurate pitot-static data, pitot-static calibration is a critical part of the test program. Further, these data are usually necessary for fly-by-wire flight control systems, angle of attack systems, stall systems, targeting systems, and any other system which requires airspeed, altitude, or vertical velocity. It should be obvious that accurate airspeed, altitude, and vertical velocity are critical to flight safety. Most modern aircraft cannot be flown without this information. In addition, the airspeed and altimeter correction of knots indicated airspeed (KIAS) to knots calibrated airspeed (KCAS) and temperature probe recovery charts in the aircraft performance manuals are developed directly from these tests.
Once calibrated, pitot-static systems are very robust. Leaks and obstructions cause the greatest problems. I have experienced leaks and obstructions as well as incipient problems caused by improper designs and miss-calibrations. All of these problems are usually detected through discontinuities between the pilot, copilot, and standby instruments. In certified aircraft, the pilot and copilot/standby instruments are required to have separate pilot-static systems.
Performance testing determines and characterizes the aircraft flight envelope. This envelope is defined primarily by airspeed, altitude, Mach, and load factor (g). Because the performance envelope is dependent on accurate airspeed and altitude measurements, performance testing can begin only after the pitot-static systems are calibrated. In general, performance testing is designed around a safety buildup, and some performance data is taken during the pitot-static calibration tests. Usually, during the pitot-static tests, the aircraft data analysis system, which records the specific test information required during flight test, is also calibrated. Often, for small changes (modifications) to an aircraft, testing is accomplished with hand data instead of instrumentation, but because of design uncertainties and the precision of the data required, this is not advisable for large modifications or new aircraft. Performance testing includes takeoff, climbout, cruise, descent, and landing.
Because it measures and defines an aircraft’s envelope, aircraft performance testing is one of the most important parts of airworthiness. Performance testing is also a key to mission suitability testing, and a large portion of analysis and M&S goes to support this part of the test program. Typically, because performance is a focus of the design process and the aerodynamic calculations, relative to other computations, are easy, this is the part computers and humans are best at estimating. Because some performance testing is hazardous, this is also a risky area to test.
Takeoff and landing flight test is the most dangerous and difficult part of performance testing. For example, testing is accomplished to determine braking distance for landings and rejected takeoffs at different representative loadings. This kind of testing frequently results in aircraft brake fires and has led to the loss of test aircraft. Rejected takeoff testing to determine ground minimum control speed (Vmcg) is also risky because the aircraft may not stay on the runway at the test end point. All performance testing is based on very carefully held tolerances. Landings and takeoffs must be flown using a specific procedure for a certain target variable while all other variables are held constant. Some endpoint testing is very difficult to accomplish because the aircraft may be “hurling at the ground” near the limit of its physical capabilities. The other areas of performance testing, climb, cruise, and descent, are more benign, but to get good data, they require the TP to hold very close tolerances. For instance, in cruise testing, the aircraft must be trimmed to maintain airspeed ± 2 knots and altitude ± 100 feet for 3 minutes. When the aircraft is stable, the TP should be able to let go and not have to touch the throttles or the control column for the entire data collection period. This is not an autopilot event; autopilots cannot hold these parameters without moving the control surfaces. The TP can’t take a long time for each point; to stay on schedule and to maximize test efficiency, the TP must get on conditions quickly—this is where the test pilots really earn their pay!
Performance data forms the primary portion of the aircraft performance manuals. These are the documents that tell aviators how much runway they need for takeoff or landing, how far they can fly before refueling, and how fast they can get there. These charts are developed for the specific altitudes, temperatures and conditions where the aircraft is operated, and they represent a critical portion of the test program. In general, these charts are first developed for the aircraft using M&S and computer derivations. The flight test then corrects these computer calculations and refines the approximations. These manuals represent the performance capabilities of the aircraft and are said to “characterize” the aircraft and define the aircraft’s “envelope.”
Aircraft performance is critical to safe flight. An experienced pilot will keep a close eye on the aircraft charted performance compared to actual aircraft performance. The recognition of a discrepancy between the two can mean the difference between landing safely on a runway or ending up in a briar patch. The usual cause of performance problems is reduced engine performance, but modifications, unauthorized paint, poorly characterized modifications, and other changes to the aircraft exterior like dents, uncovered holes, open latches, etc. can radically decrease expected aircraft performance.
Now that we know the aircraft can takeoff and land, the next step is to check the physical integrity of the aircraft. This is accomplished through structural flight test that verifies the results of M&S, estimations, ground, and lab testing. There are two distinct types of structural flight test: trials and measure tests. Trials are uninstrumented flights that verify the limits of the flight envelope. They are usually accomplished on aircraft modifications where the aircraft are to be permanently limited, there is little engineering uncertainty, or the modification has been already proven on a similar variant. In these tests, the aircraft is flown in a buildup to the edges of the envelope. The aircraft is inspected between flights to determine whether it suffered a structural failure. These types of tests are only appropriate for restricted/experimental aircraft and well-characterized modifications. A restricted/experimental aircraft is one you know will be limited because of a modification, for example, a test aircraft that will never be flown more than 80% of its normal airspeed. A well characterized modification is one where the structure of the aircraft and the modification are so well known that the margins of safety can be calculated to be above the normal structural strength of the aircraft (usually a 25% margin above the 1.5 factor of safety).
For most modifications and all new aircraft, the structural integrity is usually proven through destructive ground testing and checked with instrumented in-flight testing. Destructive ground testing is the process in which the aircraft structures are placed under load to determine their breaking points. This provides a reasonable estimate of the ultimate breaking point of the aircraft. The structural integrity is then checked in the air by measuring the stress and strain on aircraft components using a buildup process. This testing must be accomplished using an instrumented aircraft! Most structural testing is accomplished first on the ground using extensive M&S and computer analysis.
Structural flight test is accomplished by gradually (buildup) pushing the aircraft to the edges of the envelope. Mach, q, and g loading define these limits. These can be dangerous tests, but because they are instrumented, the flight test engineer can monitor the testing to protect the aircraft and crew.
These data are critical to the aircraft program and together with other data such as performance help define the full aircraft envelope. For instance, during performance testing we attempt to define the top speed of the aircraft, however, because the aircraft may not be able to physically handle some forces, structural testing may limit the aircraft top speed below the theoretical performance maximum. This is true of the T-37 where the maximum airspeed is based on the strength of the canopy.
Aircraft structural problems are generally the result of age, wear, misuse, unauthorized modifications, and improper maintenance. Structural limits define a key part of the envelope of the aircraft. Under normal use (wear), an aircraft may easily outlast its structural fatigue life (age); however, misuse, unauthorized modifications, and improper maintenance, can all lead to either decreased fatigue life or immediate structural failure. Examples of obvious misuse are over-g, hard landings, power mishandling, and overspeed. Less obvious but typical misuse are thunderstorm penetrations, turbulence, overloading, icing, snow, and poor ground handling. Unauthorized modifications and improper maintenance can result in inappropriate engineering changes, like holes and loads, where they shouldn’t be. Strict adherence to the manufacture’s instructions, certified modification procedures, and the certification inspection requirements go a long way to ensuring an aircraft’s continued structural integrity.
Aerodynamic testing is similar to structural testing. We perform uninstrumented testing on well-characterized and restricted/experimental modification aircraft, while for major modifications and new aircraft, we always perform this testing using instrumented flight test. Aerodynamic testing determines the flutter, aeroacoustic, and buffet characteristics of the aircraft. All three of these characteristics are, to some degree, dependent on dynamic pressure (q). Flutter is a very dangerous characteristic in which the airflow on a surface of the aircraft excites a structural vibration. This effect can result in the destruction of the aircraft. In many cases of flutter, a surface on the aircraft, for instance a wing or other aerodynamic surface can break off. In one case of a modern home-built aircraft, the pilot used cellophane tape to repair a cut on a Mylar aileron surface. During cruise flight, the tape came loose, caused the aileron to flutter, and the wing came off the aircraft. During an air show, flutter caused the loss of an F-117: half of the attachments for the aileron actuator structure were missing, and during a high-speed pass the aileron fluttered the wing off. Antenna flutter is also common. Aeroacoustic vibration is a localized high frequency vibration on an aircraft. It can result in damage to the aircraft, for instance, the destruction of an antenna or the skin on the wing or fuselage. Buffet is low frequency vibration caused by airflow coming off one part of the aircraft and striking another part the aircraft in a way that is not foreseen. Expected and identified buffet is usually okay.
Ground analysis, especially M&S, ground testing, and computer modeling, is used extensively to determine the flutter, aeroacoustic and buffet characteristics of an aircraft. Unfortunately, only flutter can be predicted with any accuracy. When flutter is predicted to be a possible problem, prior to flight test, a ground vibration test (GVT) is accomplished on the aircraft or modification. Aeroacoustic problems can sometimes be predicted using M&S and wind tunnel testing.
Aerodynamic flight test is accomplished in a build-up in combination with structural testing. For new aircraft or modifications, the testing must be instrumented, although for well-characterized or restricted aircraft, the testing may only be an envelope verification (fly out to the end points and see what happens). All these kinds of testing are approached with great care because the outcome can be catastrophic. The basic approach to aerodynamic testing is a controlled incremental build-up while attempting to excite the aerodynamic characteristic. For example, in flutter test, the pilot excites vibration in the aircraft using stick-raps or through an electronic stimulation system. These tests are usually accomplished at 5 knot increments up to 15% beyond the max speed of the aircraft.
This testing ordinarily defines the aircraft’s maximum airspeed. In general, the aircraft envelope must be 15% less than any flutter boundary. If flutter, aeroacoustic, or buffet problems are discovered in the normal envelope, the aircraft must be limited or redesigned.
I think you got the message about flutter. Don’t go looking for it, but on a certified aircraft it will be almost impossible to find. Unauthorized modifications and improper maintenance can produce flutter sinks, so there is even more reason to steer clear of them. Unauthorized modifications and improper maintenance are also the main cause of aeroacoustic and buffet problems. I once flew three functional check flights on a Sabreliner that demonstrated repeated rudder buffeting at high speed. Maintenance couldn’t find anything wrong with the aircraft. After the third flight, I discovered a 2 inch high by 1 inch round GPS antenna had been installed ahead of the vertical stabilizer to support a test. This little antenna caused enough rudder buffet to scare me half to death. We moved the antenna.
Handling qualities describe the way the aircraft responds to the aircraft controls. The desired outcome of all handling qualities testing is for a “carefree” aircraft that is safe and easy to fly.
Handling qualities are very difficult to predict, and a great deal of M&S, computer modeling and ground testing is done to try to forecast these characteristics. The F-5 is an aircraft in which a small change from the T-38 resulted in a significant and unpredicted handling qualities change. The T-38 has wings without a leading edge slat (type of flap) while the F-5 was designed with powered slats to improve low speed handling qualities. To accommodate the motor that moves the F-5 leading edge slat, a wing extension called a strake was added at the wing-root to encase the motor. During flight test, this leading edge extension caused the F-5 low speed handling qualities to be significantly different and improved compared to the T-38. Subsequently, leading edge extensions are used on almost all modern fighters.
Handling qualities flight test is usually accomplished through longitudinal and lateral-directional flight test techniques. All of these tests can be accomplished using uninstrumented techniques, but for major modifications, fly-by-wire aircraft, and new aircraft they should be made using instrumentation. To accomplish these flight test techniques, the pilot makes small inputs to the flight controls under specific flight conditions, and the test crew measures and records the aircraft’s response to the control inputs. In fly-by-wire aircraft, we use Handling Qualities During Tracking (HQDT) tasks in conjunction with instrumented systems to measure the aircraft’s handling qualities. HQDT are high workload flight events such as air refueling, offset landings, and pitch, roll, and heading capture tasks. A measure of success based on the aircraft’s ability to accomplish these tasks with acceptable precision and pilot workload is designed for each of these tasks.
These data are used to write the flight manuals, program simulators, and prepare for aircrew training. Poor handling qualities can send a program back to the drawing board. For instance, the wing drop on the F-18E/F is a classic handling qualities problem. This problem was caused by the leading edge strakes, the same characteristic that improved the F-5’s handling. This characteristic is as unpredictable today as it was when the F-5 was in development. If a handling qualities problem does not significantly affect the aircraft airworthiness, mission suitability, or certification the problem can be documented in the flight manual and training programs. If not, the aircraft must be modified to improve its handling qualities. The external strakes seen on the bottom and aft of the fuselage on many commercial aircraft are examples of modifications to improve lateral-directional handling qualities.
An aircraft’s handling qualities are transparent to most pilots: generally, a pilot quickly learns to compensate for the minor handling problems of an aircraft and tunes them out. Commonly, student pilots will notice poor aircraft characteristics the instructor long ago stopped thinking about. The instructor’s usual answer is, the aircraft always flies like that. The student soon compensates for the aircraft and forgets that the characteristic ever was a problem. Test pilots who flew the T-37 in Air Force undergraduate pilot training usually remember the aircraft as a good stable instrument aircraft. They are surprised when they evaluate the aircraft during test pilot school to find it is very snakey during instrument flight. The T-37 like many similar low-wing, relatively short, wing-loaded aircraft, has a tendency to Dutch roll at high frequency in the landing configuration. Every movement of the controls excites the oscillation, but if you are not looking for the motion and you have compensated for it, this poor feature of the aircraft will not be evident to you. Incidentally, on final approach, a high frequency snakey Dutch roll is a common feature of many aircraft.
Stall and Engine-Out Testing
Stall occurs when an aircraft wing doesn’t provide enough lift to support the aircraft in flight. Stall is a characteristic of g loading, weight, and airspeed. Stall and engine-out testing provides data to define the slow speed side of the aircraft envelope.
M&S and computer analysis is accomplished to predict the stall characteristics of an aircraft. Unfortunately, like handling qualities, this is a very complex engineering problem that we currently can’t answer very well. For example, the initial analysis and M&S for the inverted spin mode of the Swedish Gripen fighter was yaw stable with good rudder control power, but flight test showed the aircraft was unstable in yaw and had no rudder power! Wind tunnel and spin tunnel model testing, have been used successfully to estimate spin and stall characteristics.
Stall testing is accomplished by slowing the aircraft in various configurations (gear down, flaps down, etc.) to determine when the aircraft stalls. For multiengine aircraft, to determine the controllability of the aircraft in stall and with engines out, these tests are made with various engines shut-off. Engine-out testing and stall testing can be very risky. Except for basic approach to stalls, these are the last tests performed, and, next to performance testing, they have the greatest potential risk. Because of this, these tests are the most carefully planned points in the test program. The reason stall testing is dangerous is because there is always the chance the aircraft will “depart controlled flight,” meaning the aircraft becomes uncontrolled and may not be recoverable. Fighter, trainers, and single engine aircraft are usually tested to the full extent of their stall and departure regimes. Large and multiengine aircraft are rarely tested beyond initial stall.
Stall and engine out testing data are used to build the warnings, cautions and stall/engine out charts in the flight manual. Stall testing is also used to define takeoff and landing speeds. These data are critical to the crews for the safe operation of the aircraft. In many aircraft, flight control and warning devices are incorporated to prevent and warn of impending stalls.
Engine-out and stall are usually the least favorite topics for aviators. In normal conditions, in a certified aircraft, unless something goes wrong, you shouldn’t see either condition. Unfortunately, every now and then something does go wrong in aircraft. Still, a certified aircraft will recover from a stall and fly with an engine out. The critical situations are those where the aircraft’s performance characteristic are different from normal. For stall, this means anything that disrupts the flow over the wings, tail, and yes, fuselage. Icing tests on the C-130J showed that as small a disruption on a wing surface as sandpaper could cause significant changes in stall characteristics. The best advice is to be wary of conditions and circumstances that change an aircraft’s aerodynamic properties. For engine-out, the best advice is to know and fly the aircraft in accordance with its certification criteria. Using those procedures will give you the greatest margin for success.
Engine testing is accomplished to determine the envelope of the aircraft engines. Usually, it is a good idea to design the engine envelope to exceed the aircraft envelope, including the restart envelope of the engine. Engine testing includes basic operation, shutdown, restart envelope, and adverse conditions such as rain (water ingestion) and icing. Engines are certified individually and when installed in an aircraft.
Much analysis and M&S is accomplished on the engines, both in the aircraft and for the engines by themselves. Ground tests include static power and environmental testing. However, because it is difficult and expensive on the ground to reproduce the conditions of flight like altitude and airspeed, ground engine testing, historically, does not tell us enough about the engine/aircraft combination.
Basic engine operation testing naturally takes place during structural and performance tests. Specific engine tests are also accomplished by shutting off an engine at the different envelope limits and comparing ground analysis to the reaction of the engine and aircraft. For single engine aircraft, this can be risky testing. Also, restarts of the engines are made at the different envelope limits to determine the “restart” envelope of the engine. Engine testing is an inseparable part of basic aircraft envelope testing.
Engine test data are used to build the flight manual restart envelope charts and to warn crews of potential engine envelope limits. For instance, the T-38 is sensitive to flame-outs (engine stalls, where the engine stops running) at high altitudes. The flight manual warns the crew to only move the throttles one at a time and at a slow rate when at high altitude.
I have seen about everything an engine can do in flight including burn. The last fire I had was while turning final approach in a C-130E at night. The loadmaster said, “Pilot, number four is on fire.” The only reply I could think of was, “you’re kidding.” We made the landing a full stop and greeted the firetrucks. Engines get a bad rap because they’re expensive, and during flight, they sometimes stop running. A certified engine with all its inspections is about as dependable as you can get, but uncertified parts or unapproved modifications or maintenance can cause engine problems. These will not enhance the dependability of your engine.
Modern aircraft are normally designed with the capability to fly and land in instrument conditions. Instrument conditions are where due to night, clouds or other conditions, the ground and/or horizon is not visible. Three types of instruments are installed in instrument aircraft: control instruments (attitude indicator (ADI) and thrust), monitoring instruments (altimeter, airspeed indicator, vertical velocity indicator), and navigation instruments (GPS, ILS, INS, VOR, TACAN, and/or NDB). The pilot uses these instruments to keep the aircraft in the correct attitude and to navigate along a route of flight. The cockpit is designed to enable the pilot to fly safely during instrument conditions. Flight test is used to certify the capability of the instruments and to determine if there are any problems with the design, display, and function of the instruments and displays. The aircraft monitoring instruments are tested during the pitot-static and engine phases.
The primary method used to test instruments on the ground is through crew systems mockups and flight simulators. These ground methods are critical to the aircraft design program, but they don’t completely predict the outcome of instrument flight test. Ground simulation is necessary to the program because it allows the development of the aircraft instrument system itself.
Flight test of instrument systems is accomplished by using the instruments in flight while recording the aircraft flight parameters. These flight parameters are then compared with “truth” data from Differential-GPS, radar telemetry, high-speed photo/video telemetry, etc. Corrections to the instruments are made from the data taken during flight test. Uninstrumented flight test can also be accomplished for simple modifications and restricted aircraft. Any aircraft, which will fly in actual instrument conditions, requires a strict test program to certify the instrument flight capabilities of the aircraft.
As mentioned, these data are used to make corrections to the instruments and instrument systems, and to generate warnings, notes, and cautions for the flight manual.
Problems with instrument systems must be corrected before an aircraft can be certified. The C-12 (Kingair) had instrument problems that required a full redesign of the instrument systems. In addition, MC-130H problems with the flight director led to extensive rewrites of the operational flight software. The time to discover an instrument problem is not when you are surrounded by clouds. Close attention during visual flight can alert you to potential problems with instrument systems in the air and on the ground. For example, on final approach, a consistent 5 degree variance on a horizontal situation indicator (HSI) from the published course can indicate a failure of the HSI, the instrument system directing the HSI, or the signal coming into the instrument system. Any of these problems is worth a check of the instrument system and if that checks out normally, a report on the navigation aid.
Systems test is the most common type of testing today. Systems test is the testing of specific, usually missionized, systems on an aircraft. For example, instrument, terrain following, radar, and electronic warfare testing are only a few of the many types of systems testing. In systems test, the aircraft can never be separated from the system that is tested. In the case of terrain following radar testing, for instance, the handling qualities, performance, displays, and engine-out characteristics all affect the flight test and design of the aircraft. Because the pilot’s perception of the world (situational awareness) is critical to flight safety and aircraft handling qualities, we have discovered that something as seemingly simple as the displays in the cockpit can significantly affect an aircraft’s airworthiness. With this in mind, systems test should never be approached without due consideration for the overall aircraft system. For instance, the installation of an oven on an aircraft resulted in the failure of the instrument systems, and in the C-131, the design of the compass system may have caused three aircraft to explode in flight.
Significant analysis, M&S, and ground tests should always accompany systems testing. Usually, this is accomplished in a mockup or simulator. These types of ground tests can tell us a lot about the system and will aid significantly in the development, fabrication, installation, and flight test of the system. By providing a statistical and analytical framework for the systems being tested, ground testing can significantly reduce the risk and scope of the flight test program.
Systems flight test is similar to instrument testing. It is accomplished by operating the system in concert with other aircraft components on the ground and during flight. The first step in any systems test is electromagnetic interference (EMI) and electromagnetic compatibility (EMC). As described earlier, these tests prove the new system will not interfere with the existing aircraft systems and are part of the basic airworthiness testing of the aircraft. Throughout the flight test program the testers watch for EMI/EMC problems. Next, we test system failure states that could adversely affect aircraft flight. Failure state testing is the most important part of systems testing. For example, the greatest concern in testing an autopilot is what happens when the autopilot fails and commands the full deflection of an aircraft control surface. Finally, the system is tested in its specific mission role.
Systems test is used to update and write flight manuals on the system’s use. It is also used in all three of the types of test criteria, airworthiness, mission suitability and certification that we already discussed. Systems testing is usually characterized by a confidence interval–the 95% and 99% already mentioned.
The systems installed on your aircraft can radically affect safe flight. As already discussed, interference is one of the largest bugaboos for electronic systems, but as systems and aircraft become more and more integrated, interference may start to look like a small problem. Software in integrated systems is very difficult to test. As anyone who has used a PC knows, lock-ups and errors are not uncommon, and these software problems can be hazardous in aircraft systems. For instance, can you stand to have your ADI lock-up while flying instruments in the clouds? Expand this example to every electronic system in an aircraft, and identify each integrated component. Every software interface has the potential to cause a system failure. This is daunting to test, and the FAA and aircraft manufactures can’t expect aviators to be their beta testers. The point is, that manufacturers go to extremes to ensure their equipment works better than advertised, but it’s possible to find unexpected problems—especially in software. Be aware.
I need to mention one special area in systems test: instrumentation systems. Instrumentation systems are the systems used to record, display, and transmit flight test data. These systems are usually required by and critical to flight test programs. The manufacturer and flight testers must take extraordinary care that these systems are airworthy, suitable for the test mission, and properly calibrated. These systems must go through analysis and flight test just like any other system. For any test program, a faulty, dangerous, or uncalibrated instrumentation system can render all the flight test data worthless. Because they record the “truth” data by which we evaluate an aircraft, these systems, when improperly tested, can add risk rather than reduce risk to a test program.
Summary and Conclusions
I covered the major types of flight test and the consequences when testing demonstrates a problem. You can see that flight test is neither a simple nor an undisciplined science. In most cases, M&S, analysis, and ground testing are an indispensable part of test preparation. You can also see that though these methods prepare for flight test, they are incomplete without flight test. All of the types of flight test support the three objectives of flight test: airworthiness, mission suitability, and certification. The goal of test and evaluation is to provide aviators an aircraft that is safe, easy, and fun to fly in the role for which it was designed.