NOUVELLES CONSULTANTS
Software and Product
Safety
EG2401 Project 7 Final Report
Muhammad Ahad U084075R
Muhammad Nadzri Hussain U096036E
Ruben S/O Sukumar U084705E
Farhan Bin Mohamad U090387B
3/10/2011
An analysis of the technical, security and ethical issues involved in the Ariane 501 Rocket and Patriot
Missile Defence System failures that were caused by software malfunction.
EG2401 Project 7 Final Report Tutorial 308
Page 1 of 17
1.1 - Ariane 501 case
1.1.1 Case overview
On June 4, 1996, the maiden flight of the Ariane 501 launcher ended in a
catastrophic failure. About 40 s after initiation of the flight sequence, at an
altitude of 2700 m, the launcher veered off its flight path, broke up, and
exploded. The accident and the inquiry report claimed that there was a
complete loss of guidance and attitude information, 37 s after the start of the
main engine ignition sequence and 30 seconds after lift-off.
1.1.2 Immediate Follow-Up Action
Immediately after such a disaster, the engineers from the Ariane 5 project
teams of Centre National d’Etudes Spatiales (CNES) and Industry
immediately started to investigate the failure. Also the Director General of
European Space Agency (ESA) and the Chairman of CNES set up an
independent Inquiry Board to examine the cause of the failure. Investigations
were carried out on the causes of the failure, the systems supposed to be
responsible, and any failures of similar nature in similar systems, and events
that possibly could be linked to the accident.
1.2 - Factual Issues
1.2.1 General Failure Analysis
Consequently, we will we be analysing the number of systems that failed,
the causes and conceptual issues regarding the causes of failure. The
investigation of its flight from the point of take of has to be thoroughly
assessed. The sequence of technical data that was traced from the flight data
records and the events that caused the failure are summarised in the table
below.
System Event
SRI and failure of the back-up Inertial Reference
Active SRI followed by the failure of the active Inertial Reference System
Solid
boosters
swivelling into the extreme position of the nozzles of the two solid
boosters
EG2401 Project 7 Final Report Tutorial 308
Page 2 of 17
Vulcain
engine
failure caused launcher to veer abruptly
Rupture of
links rupture of the links between the solid boosters and the core stage
Self-
destruction
of the
launcher
due to the rupture of links
Table 1: Information extracted from (LIONS, 1996)
Hence, considering the sequence of events it was evident the root of the
problem or the initial error that triggered the sequence of failures was the
flight control system which is characterised by the SRI as it failed to function
accordingly.
1.2.2 Function of SRI
The function of the SRI was primarily for the purpose of measuring the
launcher characteristics and its movements in space. Having an independent
control system (internal computer) which in turns calculates angles and
velocities based on information from a "strap-down" inertial platform, with
laser gyroscopes and accelerometers. The data from the SRI to the On-Board
Computer (OBC) executes the flight program and controls the nozzles of the
solid boosters and the Vulcain cryogenic engine, via servo-valves and
hydraulic actuators. (LIONS, 1996). Two SRIs work in parallel and there are
also two OBC to ensure reliability and at any time only one is active and the
other is in standby. If failure is detected in a single system, it automatically
switches to the other unit only if it is functioning properly.
1.2.3 Why SRI Failed?
The software design which was used in SRI in Ariane 4 (An earlier version
of the launcher) was reused in the SRI design in Ariane 5. However after the
investigation it was discovered that the software error occurred in the active
SRI and the main control system did not switch to the other SRI due to the
similar software problem. The internal SRI software error was caused during
EG2401 Project 7 Final Report Tutorial 308
Page 3 of 17
execution of a data conversion from 64-bit floating point to 16-bit signed
integer value. The floating point number which was converted had a value
greater than what could be represented by a 16-bit signed integer. This caused
a command error which resulted in internal alignment function known as
horizontal bias (BH) which is related to the horizontal velocity sensed by the
platform. This resulted in the shutdown of both SRIs.
1.3 - Conceptual Issues
1.3.1 Culture of the Ariane program
The ethics behind the software design is extremely complex to deal with.
However the one of the cause could be the culture of the Ariane Program
which only addressed the random software failures which could be efficiently
dealt with by back-up systems. The design of the software was such where the
2 SRIs would shut down when an exception was raised is an unnecessary
function in the software. On an ethical context, this type of failure should have
been simulated even before failure. The process creating such complicated
software involved addressing problems such as possibility due to failures.
However according to the inquiry report a bias nature was adopted in
solving of random failures. Since the supplier of the SRI was only following
specifications or design requirement provided by the host company, which
stipulated that in the event of any detected exception the processor was to be
stopped. However the exception which occurred was not due to random
failure but a design error. Even though the exception was detected it was
inappropriately handled because the view had been taken that software should
be considered correct until it is shown to be at fault. The board has reason to
believe that this view is also accepted in other areas of Ariane 5 software
design. After complete investigation the inquiry board was in favour of the
opposite view, that software should be assumed to be faulty until applying the
currently accepted best practice methods can demonstrate that it is correct.
(Aviation Week and Space Technology, 1996)
EG2401 Project 7 Final Report Tutorial 308
Page 4 of 17
1.3.2 No proper testing done
This is a grey area that needs to be addressed. Years of testing were done,
but much emphasis was only hardware failures and problems. Due to the
critical nature of the software in space launchers, proper modelling of software
design is vital to prevent disasters. The extensive tests conducted during the
Ariane 5 development program did not include proper analysis and testing of
the inertial reference system or of the complete flight control system, which
could have detected the potential failure. Even though it is not possible to test
the SRI as a ''black box'' in the flight environment, unless one makes a
completely realistic flight test, it is still possible to do ground testing by using
simulated accelerometer signals in accordance with predicted flight
parameters, while also using a turntable to simulate launcher angular
movements. Such a simple and accurate test been could performed by the
supplier or as part of the acceptance test. This would have exposed the failure
and the problem would have been rectified. (Aviation Week and Space
Technology, 1996)
1.3.3 Adequate Software Review
From the inquiry report, it is evident that no proper software review was
done prior to launch or hardware simulations. No stringent rules or restrictions
were in place to ensure that software was up to standard and met requirements
for further testing. For such space missions one should note that thousands of
functions can be correctly performed and one mistake can be mission
catastrophic. However mistakes can be prevented by oversight, test, and
independent analysis. A proper software review was much needed to avert
such a failure.
1.3.4 Management and Organizational Factors
The coordination between supplier of SRI and Ariane 501 team and also
among Ariane 501 team has to be questioned. Furthermore, the Ariane 501
accident report is almost totally silent about organizational structure problems
as it does not describe the allocation of responsibility and authority for safety
nor does it mention any organizational or management factors that may have
influenced the accident. Proper procedure manuals were absent and problems
EG2401 Project 7 Final Report Tutorial 308
Page 5 of 17
that surfaced might have been ignored by senior aviation specialists. The
reason behind this lack of communication remains a mystery and is hard to
prove or totally dismiss that such problem existed. From the examination of
accident data, there were signs the initial problems were not properly dealt
with by the Ariane 501 team.
Also, a more transparent organization of the cooperation among partners in
the Ariane 5 would have averted the disaster. With excellent cooperation
among team players, together with clear cut authority and responsibility, is
vital to achieve system coherence. In most similar magnitude disasters,
inadequate transition from development to operations was the key reasons
cited for failure. Engineering management has at times sometimes has a
tendency to focus on development and to put less effort into planning the
operational phase.
1.4 - Ethical Issues
The ethical issues for this case have to be carefully examined to identify mistakes,
irregularities, weaknesses in management, development and operation problems and
the overall co-ordination. Issues will be examined with due regards to software failure
and software ethics. The cost of development of Ariane 501 which was around $7
billion and the total lost was around $370 billion US dollars and this makes the
financial lost is extremely drastic. Hence problems will be addressed form a ethical
point of view.
1.4.1 Utilitarianism
From rough cost benefit analysis, it should be noted that the cost of
developing proper software involves around 2 years, involving a lot of team
players which translate to man ours, and financially costing around billions.
The benefits here are not short term, but considered to be long term.
Reputation is also at stake here and hence it is evident that cost-benefit
analysis cannot be reduced to just dollars here. So when the engineers who
designed the software by implementing the same software of Ariane 4 on the
Ariane 5 without evaluating all possible aspects or scenarios that will result
EG2401 Project 7 Final Report Tutorial 308
Page 6 of 17
due to this. Under the Rule Utilitarianism approach, certain good practices
were broken and this resulted in a bad design of the software.
1.4.2 Duty ethics
One might come to the conclusion that duty ethics is not of mush
importance as there was no fatality. However the software designers had a
professional duty to fulfil. They had a task of producing complex software for
the engineers who were working with the hardware design. Since the inquiry
report had clearly stated that proper testing of the software was not done
efficiently to determine potential problems, the duties of the software
engineers are seriously flawed. Apart from the software engineers, the co-
ordination of key Ariane 501 team players was lacking and everyone had a
responsibility to work together to ensure that the launch was successful.
Communication breakdown together with failure to exercise proper
responsibilities have illustrated the importance of duty ethics. The duties of the
higher management come into serious question. They should have initiated
proper software reviews for the Ariane 501which could have detected the error
at the start. Every engineer in the 501 program has the right and duty ensure a
successful launch.
1.4.3 Virtue Ethics
Certain actions of engineers and professionals do not reflect good character
traits. Loyalty to the profession is an ethical code all engineers should follow.
Apart from this, loyalty to employers is also a character trait that successful
engineers should have. Such qualities were absent from the Ariane 501
industry partners and engineers. Negligence and careless nature of the
software design team was reflected in the post- accident analysis by experts.
Such a negligent attitude has cost billions and has left a huge dent in the
Ariane reputation.
1.4.4 Self –Realization Ethics
The aspect Ethical egoism is also an issue here. The idea of long-term well-
being rather than a narrow, short-sighted pursuit of immediate success or
pleasure have to be examined carefully. From the whole process from research
EG2401 Project 7 Final Report Tutorial 308
Page 7 of 17
to development, testing and launching, it can be concluded that the search or
mind-set for immediate success was present among top management. Hence
careless or negligent attitude might have resulted from this trait. Such a rush to
ensure success has caused such a horrendous catastrophic failure.
1.5 - Recommendations and current practices
Based on the above analysis and the inquiry report following recommendations would
have helped to minimize the possibility of the Ariane 501 disaster.
The alignment function of the inertial reference system should be switched off as soon
as lift-off occurs. It should also be ensured that no un-necessary software runs during
flight. The system should be thoroughly tested by using as much real data as possible.
Also, flight simulations should be conducted before any mission. All sensors should be
engaged and send their best captured data. Thus, no sensor should be allowed to stop
sending its best effort data. Furthermore, specific software qualification reviews should
be conducted. Industrial Architects or whoever is in charge should participate in such
reviews and report on complete system testing together with equipment analysis.
Additionally, all flight software (including the embedded software) should be
comprehensively reviewed.
o This includes checking the assumptions (explicit and implicit) in the software code
against the constraints on equipment usage.
o The range of values taken by variables in the software should be realistic.
o Potential problems in the on-board computer software should be proposed by the
project team and then reviewed by a group of external experts.
Also, whenever it is technically possible, exceptions to tasks should be confined and
backup capabilities should be devised. More data should be provided to the telemetry
when a component fails, so that it would be less essential to recover the equipment. The
definition of critical components should be reconsidered, by taking failures of software
origin into account, especially single point failures.
External parties should also be consulted while reviewing specifications, code and
justification documents. Such parties should not be involved in the project and should
EG2401 Project 7 Final Report Tutorial 308
Page 8 of 17
consider the substance of arguments rather than check that verifications have been carried
out. The trajectory data should also be included in specifications and test requirements.
The test coverage of existing equipment should be reviewed and extensions should be
made where necessary. The justification documents should be regarded as important as
the code and techniques for ensuring consistency between code and its justifications
should be consistent.
A team should be set up which would prepare the procedure for qualifying software,
propose stricter rules for confirming such qualifications and ensure that the specification,
verification and testing of the software are of a consistently high quality in the Ariane 5
programme. This should also include external experts.
Lastly, there should be a more transparent organization of the cooperation among the
Ariane 5 programme partners. Engineers should collaborate very closely and there should
be clearly defined roles of authority and responsibility to ensure system coherence.
1.6 – Conclusion
The Ariane 501 failure is a multi-faceted example of how software errors, lack of
adequate testing, maintenance and coordination can lead to big disasters, resulting in a
multi-million dollar loss.
There are several issues involved in the failure. The usage of the old specifications
from Ariane 4 was not compatible with Ariane 5 (since its path was considerably
different). The testing performed was quite inadequate and did not take into account many
factors involved. The software-equipment compatibility was also not tested to as far an
extent as possible. The various parties involved in the project did not have a clear,
transparent communication interface. Thus, all of these issues led to the failure of Ariane
501. We saw that eagerness to achieve success can often lead to short-sightedness and
make organizations overlook essential safety precautions and thorough testing which can
result in disasters like this. Hence, it is important to adopt a strategy of comprehensive
error testing, maintenance, coordination between teams involved in the project and
ensuring that different system components are highly compatible. In order to achieve
these objectives, it is also very helpful to consult third party or non-partial experts that
can review and further check the system for any discrepancies.
EG2401 Project 7 Final Report Tutorial 308
Page 9 of 17
2.1 - Patriot MDS case
2.1.1 Case overview
In 1991, a war in the Middle East broke out commonly known as the Gulf
War. The war was led the Americans and Britain against Iraq, the army of
Saddam Hussein. Operation Desert Storm was the name given for the battle
occurred in the Gulf War mainly because of the operations and military
participation.
In February 25th
, 1991, a Patriot missile defence system, failed to track and
intercept an incoming Scud missile during the Operation Desert Storm in
Dhahran, Saudi Arabia. This failure claimed 28 American soldiers in an Army
Barrack and is considered to be one of the worst software failures in history.
2.2 - Factual Issues
When the Patriot missile defence system was first invented, it was originally meant
to operate in Europe. It was used as a defensive measure against the Soviet’s cruise
missile and medium to high altitude aircraft missiles, which were able to travel up to
twice the speed of sound. The Patriot system provides good mobility so as to prevent
contact. Because of its inconsistent location, it can only be operated for a few hours
before moving on to its next location.
In Operation Desert Storm, the Patriot missile system was deployed for the very
first time against the Scud missiles. The scud missiles, they can reach speeds of up to
five times the speed of sound. Scud missiles were new to the Army and any data
regarding the missile were very scarce. The Army had to rely a lot from external
sources especially from the intelligence agency. Without proper knowledge or data of
the Scud missile would make the Patriot missile system irrelevant. Therefore it was
very important that the Army had to understand how tracking and intercepting the
Scud missiles could be achieved effectively.
According to reports “The Patriot Battery at Dhahran failed to track and intercept
an incoming Scud missile because of a software problem in the system’s weapons
control computer”. This resulted in an inaccuracy in the calculation that worsened as
EG2401 Project 7 Final Report Tutorial 308
Page 10 of 17
the system operated longer. Based on findings, the battery has been operating for 100
hours. And by then the inaccuracy was so serious enough to cause the system to look
in the wrong place for the incoming Scud missile.
The irony was that the Patriot defence system had never been used before to
defend against Scud missile nor was it able to operate in long operating hours.
Reports indicated that Army officials received a data from Israeli two weeks before
the incident. The data indicated some loss in accuracy after the system had been
running for 8 hours. However, modifications were made to the system to improve the
software but it did not reach Dhahran until February 26, 1991. The day after the Scud
missile hit the Army Barrack.
2.3 - Conceptual Issues
The weapons control computer is actually the most important lifeline for the
Patriot missile system. However, the Patriot missile system that was used in Operation
Desert Storm was designed back in 1970s. This shows a very huge limitation in
precision tracking and intercepting, which is the main purpose of the missile.
To understand how radar works, it sends out electronic pulses in the air that is
surrounding it. These electronic pulses when hit a target would bounce back to the
radar thus giving information such as distance and location. It is similar to the
working principle of the Patriot’s missile system. An additional feature to the system
is the ability to select specific missile types that is required for the operator. And for
the case of the Patriots missile system, Scud missiles were selected as the target of
choice. The system's range-gate algorithm is the important factor in the accuracy of
the missile’s position.
When a flying object has been detected by the Patriot’s radar, the range gate area
would calculate only the area that the object is flying and filters out anything that’s
outside the area. Only when the object is in the area and calculated, then can it be
confirmed that the object is a scud missile. The range gate area requires that the
missile be exactly in the centre. And it follows the same throughout where the system
would next look out for it.
EG2401 Project 7 Final Report Tutorial 308
Page 11 of 17
Data came running through on 11 February 1991, that there was a 20% shift in the
radar range gate area after the system has been running for 8 hours consecutively.
This shift meant that the target was not in the centre of the range gate area and it
proves to be a hazard for the system. A shift in target would mean that the radar could
lose track of the missile and no longer intercept. As stated in the incident report
“Patriot Project Office officials said that the Patriot system would not track a Scud
when there is a range gate shift of 50% or more”. Based on simple calculations a 50%
shift in the range gate area would approximate to about 20 hours of continuous
operations. Basically in conclusion, running the Patriot system for more than 20 hours
would spells danger as the inaccuracy becomes too great. Thus, this results the radar
to not be able to detect the scud missile. For the case on the 25 February 1991, the
Patriot radar system has been running for 100 hours consecutively and has shown how
fatal and deadly the system could be.
2.4 - Ethical Issues
Assumption made by the army officials that the Patriots were not running their
systems for more than 8 hours at a time was one of the key factors that caused the
disaster. However, when the Israel data was analysed and loss in targeting accuracy
was detected, the officials made software changes which rectify the inaccurate time
calculation. The change made it possible for the system to run longer and was part of
the modified software version that was released on February 16, 1991
On February 21, 1991, the Patriot users received a message from the Patriot
Project Office. It was stated that running the system for a ‘long’ period would result in
a shift in the range gate, which would cause in the target being offset. It was also
stated in the message that software update was being sent that would amend the
system targeting. On the other hand, the message did not specifically narrate the
definition of very long run times. This was because the army officials assumed that
the users would not run the system batteries continuously for such a long period of
time which would cause the error to emerge. This leads to the army official not giving
sufficient information as they though it was not required.
EG2401 Project 7 Final Report Tutorial 308
Page 12 of 17
Unfortunately after the Israel data was received, immediate actions was not taken
by the Army officials to determine the period that the Patriot missile could operate
before the inaccurate time calculation would cause the system to be ineffective. This
caused the modifications being sent out to the system later than expected.
Ethical issue is the major root of this disaster as it leads to a lot of other problems.
Software systems can be more complicated therefore it needs very thorough review
process and testing. The engineers failed to discover all bugs as although it is difficult
to test all combinations of inputs or variable values. They also did not ensure that the
rectification was done immediately. The complacency that the system will not run for
more than 8 hours was also one of the ethical issues. This shows that software
engineering principles may not be adhered properly.
2.5 - Analysis of the Ethical Issues
From this incident, a few ethical issues can be discussed. Although the main cause
is in the numerical error, there were a lot of events that led up to the disaster. This was
hugely contributed by ethical issues. As a result, it caused a catastrophic failure
involving loss of human life.
Inconsistency was one of the primary factors of this mishap. When the error was
discovered at Raytheon, the amendment was only made to some areas of the codes.
This caused the inconsistency in the codes as they were not operating in the same
function. The updated codes were running with a faster and more precise function.
This program was developed fifteen years ago in assembly language. With updates
and new codes added, it made the system tough to operate and maintain. Therefore the
defect was not identified before the incident.
In addition, complacency was another cause of the disaster. The army officials did
not bother to investigate the number of hours that the system will be running. By
assuming that the systems will not run for a very long time continuously caused the
message received without any detailed specification of when the system will incur
error.
EG2401 Project 7 Final Report Tutorial 308
Page 13 of 17
Another factor that contributed to the software engineering ethical issues is
maintainability. It is very important to maintain an old system. In 1986, when the
Patriot Missile System was modified to enhance missile tracking, the developers
chose to recode the software in a high-level language. This would improve the system
performance and maintainability. The fifteen year old assembly code that constantly
patched was too complicated for the programmers to understand. This caused them
the inability to do a better job when dealing with the system. The checks for a critical
safety system should be done comprehensively. It should have been detected through
vigorous debugging and unit testing, though it can be quite intricate and expensive.
The problem could also be identified if retesting is done after every update or
patching.
Reaction time taken by the army officials to rectify the software problems was not
fast enough. The officials should have reacted with much prompt and effectiveness.
This delay in the response leads to the disaster as the message was not received on
time. An ethically responsible software engineers would have taken the matter
seriously and reacted accordingly to prevent this mishap.
In addition to the bad timing of the bug fix’s late arrival, it causes the loss of life in
Dhahran. Twenty-eight soldiers lost their lives while serving their country. Had better
care been taken to conform to ethical standards, their deaths might have been avoided.
2.6– Recommendations
Ethical Responsibilities should be instilled in all individuals dealing with the
system software. They should avoid falsifying test results or covering up known or
suspected problems. Leaving potentially serious bug in a product and violating safety
standards. When adapting an older software system to a new user, the new user should
understand the whole system before operating on them.
There must be sufficient analysis of potential failure modes and consequences. To
design and implement appropriate safety and robustness features. They should have
reacted aggressively to reports and warnings after the possible problems.
Engineers and management should not underestimate the complexity of the
software. Poor software engineering techniques should not be use given that there was
EG2401 Project 7 Final Report Tutorial 308
Page 14 of 17
little to no documentation available regarding the developments of the software. The
management should react faster to communicate the problems with other users. Most
importantly protect against error and not error discovery.
In making software engineering a beneficial and respected profession, software
engineers shall commit to making design, analysis, testing, specification, maintenance
and development. Software engineers shall follow these following Eight Principles, in
accordance to their commitment to the welfare, health and safety of the public:
1. PUBLIC - Software engineers shall act consistently with the public
interest.
2. CLIENT AND EMPLOYER - Software engineers shall act in a manner
that is in the best interests of their client and employer consistent with the
public interest.
3. PRODUCT - Software engineers shall ensure that their products and
related modifications meet the highest professional standards possible.
4. JUDGMENT - Software engineers shall maintain integrity and
independence in their professional judgment.
5. MANAGEMENT - Software engineering managers and leaders shall
subscribe to and promote an ethical approach to the management of
software development and maintenance.
6. PROFESSION - Software engineers shall advance the integrity and
reputation of the profession consistent with the public interest.
7. COLLEAGUES - Software engineers shall be fair to and supportive of
their colleagues.
8. SELF - Software engineers shall participate in lifelong learning regarding
the practice of their profession and shall promote an ethical approach to
the practice of the profession.
The role and responsibility of a software Engineer is to act professionally and
ethically by understanding and appreciating risk factors. They are to ensure proper
attention is paid to these principles
EG2401 Project 7 Final Report Tutorial 308
Page 15 of 17
2.7 - Current Practices
Current situation for the Patriot Missile would be to restart the system for 2 to 4
hours. Turning off and on the system helps the system to refresh itself and reinitialize
the computer time back to zero. To reboot the system it would take only less than a
minute.
Another immediate action taken by the Army would be to provide advanced
training for the Patriots maintenance operator. Raytheon has developed several
training to take the highly experienced ones to a much higher level of maintenance
operator. Every update in the software or upgrades is being taught to the maintenance
operators. Thus in time, there would be many highly trained and skilled soldiers.
Lastly though still under prototype, conventional radar has been replaced by an x-
ray radar system. Unlike the conventional radar, it do not use electrical pulse or radar
waves to detect a missile but instead uses x-rays that is much more superior. Hence,
the name x-rays radar system, Therac-12.
2.8 - Conclusion
In conclusion, we realized how much people these days rely on such software or
computers to help us protect ourselves or even help to save lives. The responsibilities
placed on such small software or computer is so great, thus needed a great care on the
software or computers. It is important to do regular maintenance and checks on the
system that everything is running fine and nothing peculiar in the software that can
result in the malfunction or the ability to not function properly.
Due to the reason that software is very prone to bugs and viruses, it is also
important to debug the software and do modifications or updates on the system. For
example, how Apple or Microsoft conducts regular updates on their software, and if
there were any bugs or viruses they would send updates or upgrades for downloading.
EG2401 Project 7 Final Report Tutorial 308
Page 16 of 17
References
Bibliography (n.d.). Retrieved from http://www.engineering.uiowa.edu/~ece_036/Lecture/Ethics.pdf
Arnold, D. N. (1996, September 24). Two disasters caused by computer arithmetic errors.
Aviation Week and Space Technology. (1996, September 12). Ariane 5 Report Details Software
Design Errors.
LIONS, P. J. (1996, July 19). Flight 501 Failure. Retrieved March 1, 2011, from
http://www.di.unito.it/~damiani/ariane5rep.html
Lum, A. (n.d.). Patriot Missile Software Problem. Retrieved from
http://sydney.edu.au/engineering/it/~alum/patriot_bug.html
Math News. (2003, October 17). New anti-missile system declared worst engineering disaster EVER .
Retrieved from http://www.mathnews.uwaterloo.ca/Issues/mn9303/arienne.php
Raytheon Company. (n.d.). Raytheon. Retrieved from "Gulf War" :
http://www.pbs.org/wgbh/pages/frontline/gulf/weapons/raytheontext.html
The Importance of Ethics in Mission and Safety Critical Software Engineering. (n.d.). Retrieved from
http://cseserv.engr.scu.edu/StudentWebPages/uletran/uletran_FinalPaper.htm