Embed Size (px)
Citation preview
That, in a nutshell, is the view of Ross Anderson, professor of
security engineering at Cambridge University's Computer
Laboratory. Anderson reckons much of the IT business is governed
by network economics. This means that as more people use a
system, it becomes more valuable. Secondly, although the cost of
the first product may be very high, copies cost almost nothing;
thirdly, the cost of switching to an alternative product may be very
high, hence winners tend to take all.
Studies elsewhere suggest that first movers in high tech markets
keep up to two-thirds of the market once that market matures.
There is thus a massive incentive to get a product to market first,
even in a debilitated condition. This intensifies when developers
don't suffer directly when their product fails. Programmers feel no
pain when the system crashes; on the contrary, they get another
chance to justify their existence.
"The real driving forces behind security system design usually
have nothing to do with such altruistic goals (as protecting end
users from privacy violations and fraud). They are much more
likely to be the desire to grab a monopoly, to charge different prices
to different users for essentially the same service, and to dump
risk. Often this is perfectly rational," says Anderson. This at any
rate is the case for "off-the-shelf" software, as opposed to software
that is produced for use in specific, perhaps individual, situations.
Com put er-enab led k i l l i ng
But before we go further, let's make
a distinction. Not many people die
when Windows reveals its "blue
screen of death" to tell you there
has been a fatal error. But when
target identification systems make
mistakes it's another story. A study
by the US Joint Chiefs of Staff
found that target misidentification
was responsible for up to 97% of
friendly-fire incidents. Historically
10 to 15 per cent of US casualties
are due to friendly fire, but in the
first Gulf war this rose to 24% of those killed in action and 17%
of the wounded. Friendly fire destroyed more than 75% of all
coalition combat vehicles lost in that war.
And when the automated fail-safe procedures at a nuclear power
plant are switched off, the fall-out can be long term. The
Chernobyl area will be unsafe for humans for the next 600 years.
On a lighter note, the plot for the popular movie The Italian Job
depends on crashing the traffic control system to allow the Mini
Coopers to escape in the ensuing chaos.
So let's be clear: when human safety is at risk, the commercial
consequences of bugs are very significant, (e.g. involving large-
scale recalls of consumer products), or when a high degree of
repeatability is required in predefinable circumstances, a different
standard applies. This quest for perfection is behind the Centre for
Software Reliability, a 20 year-old project to discover ways to build
more reliable systems with fewer bugs at lower cost. Most of the
work is split between computer labs at the City University,
London, and Newcastle University.
Feel t he pain
Cliff Jones, research director at Newcastle's School of Computing
Science, also heads a CSR project on dependable computer-based
systems, a term which reflects that many large systems involve
people as well as computers. He reckons we have reached a pain
threshold. The cost of work lost or interrupted by buggy programs
or malicious attacks has prompted users to insist on better quality
software.
te
ch
no
lo
gy
38
Info
secu
rity To
day
May/Ju
ne 2
004
The rocky road to reliable software
Why are computer systems unreliable? Simply, it pays programmers to let users f ind the bugs.
Ian Grant [email protected]
Anderson: rat ional to
dump risk
The forces driving systemdesign are more likely to be
the desire to grab a monopoly,to charge different prices todifferent users for the same
service, and to dump risk.
"Last year M icrosoft stopped development on
Windows for more than a month to fix a stack
overflow bug. It must have cost them a lot to have
programmers standing around while it was fixed,
but that tells you what's at stake," he says.
Jones may be understating corporate grievances.
In M ay the US Business Roundtable, an
association of 150 chief executives whose
companies together turn over more than $3.7
trillion, issued a call for software developers and
users jointly to take responsibility for IT security.
This came after research estimated that "cyber
attacks" last year has cost the US financial sector
alone nearly $1 billion.
The Round Table also issued a statement of
principles. This includes a call to IT service
providers and software vendors to "develop and
maintain secure products and services that place
minimal burden on the end-user, while also
providing timely alerts when new vulnerabilities are
detected".
Jones says the target is to produce warrantable
software for the emerging very large, "pervasive"
networked systems. This means finding new ways
to provide fault prevention, fault removal, fault
tolerance and fault forecasting - the four basic,
complementary, technologies that are involved in
achieving highly dependable, i.e. trustworthy, systems.
Formal proof, a branch of mathematics, uses a set of stringent
techniques that leaves no room for error or uncertainty, and can
underpin both fault prevention and fault removal in particular.
The logic is closed, hard-wired. At the end, the audit sign-off
verifies that for all starting and operating conditions the system
produces the expected result, and only the expected result.
A bug's life
A version of Murphy's Law governs software reliability in that the number of defects
that survive a selection process is maximised.
This applies equally to software and to species; software testing removes the
minimum possible number of bugs, consistent with the tests applied, while
biological evolution enables a species to adapt to a changed environment at a
minimum cost in early deaths.
The behaviour of systems that contain a single bug, or a small number of them, is
known to be governed by Poisson survival statistics. For example, the probability pi
that a particular defect remains undetected after t statistically random tests is given
by pi = e-Eit. The quantity Ei is sometimes called the 'virility' of the defect, and
depends on the proportion of the input space that it affects.
The problem is that extensive empirical investigations have shown that in a large
and complex system, the likelihood that the t-th test fails is not proportional to e-Et
but to k=t for some constant k. Thus in order for software to have a mean time to
failure of (say) 10,000 hours, it must be tested for at least that much time.
This analysis gives a number of interesting results. In addition to explaining the
observed reliability growth of k=t, we show that under assumptions which are often
reasonable,
• it is the best possible
• that there are fundamental limits on the reliability gains to be had from re-usable
software components such as objects or libraries
• that the failure time measured by a tester depends only on the initial quality of
the program, the scope of the testing, and the number of tests;
• that each bug's contribution to the overall failure rate is independent of whether
the code containing it is executed frequently or rarely (intuitively, code that is
executed less is also tested less)
• and that it is often more economic for different testers to work on a program in
parallel rather than in series.
The failure time measured by a tester depends only on the initial quality of the
program, the scope of the testing and the number of tests. The tester's
measurements give virtually no further information about the quality of the program
or its likely performance in a customer's environment, where the distribution of its
input states may be quite different. A debugging team usually achieves a rapid
improvement in failure time and quickly reaches the point of diminishing returns.
However, when the ' tested' software is passed on to another tester, a number of
important bugs are often found quite quickly.
One can improve the reliability of software by using objects, provided they have
been tested more thoroughly than the rest of the software in which they are
deployed. This might be the case if they have already been extensively field tested in
other products, or obtained from a company with a more thorough test department
than one's own. However, the software's overall failure rate will then be dominated
by terms that correspond to new code, and this will limit the achievable reliability
gain.
Excerpted from Murphy's law, the fitness of evolving species, and the limits of
software reliability, by Robert M. Brady, Ross J. Anderson, Robin C. Ball. Published in
September 1999 as Technical Report No 471 by the University of Cambridge
Computer Laboratory, ISSN 1476-2986. http://www.cl.cam.ac.uk/
te
ch
no
lo
gy
39
Info
secu
rity To
day
May/Ju
ne 2
004
Jones: t rustworthy comput ing the target
Com plex syst em s
But the commercial world is more complex and unpredictable.
How many different types of document can you produce with a
word processing programme? How many alphabets and grammars
must you support? How many typefaces do you need? How many
tables, pictures, hotlinks do you need to include? How many
different printers will output hard copy? What's the power supply
like? Clearly, this is a different environment.
Even so, John Fitzgerald, one of Jones' co-researchers at
Newcastle, says Japanese colleagues have used formal methods to
develop and integrate part of a back office financial trading
system. This took between one-third and one-half of time
expected using normal development methods.
"The idea is to catch bugs earlier when they are cheaper to
correct. This leads to faster development and less need for
redesigns, particularly when you are trying to fit sub-systems
together," he says. Fitzgerald and his colleagues are about to start
work on Project Rodin. This is a new EU-funded effort to build
tools that let programmers incorporate formal methods in their
work.
Some software companies could produce what is in effect hard-
wired versions of their software. "Embedded" software is already
widely used in many applications from toys like Furby to ignition
management in cars, and in a great variety of consumer and
technical products. Such software can offer a slick, reliable but
limited set of features. Through such limitations, the overall
complexity of the software can be greatly reduced, compared to
that of a more general-purpose system, with a consequent increase
in reliability and performance. The same principle applies to so-
called RISC computer architectures; the reduced set of
instructions in the operating system results in better reliability and
faster execution of tasks. Indeed, one consultancy reckons a
stripped down version of Windows XP (normally a complex
instruction set system) will produce excellent results with less than
a quarter of the standard "footprint" of features.
The downside to embedded software is that an upgrade usually
te
ch
no
lo
gy
40
Info
secu
rity To
day
May/Ju
ne 2
004
Five minutes w ith…
Bruce Schneier is the founder and chief technical
off icer of Counterpane Internet Security, w hich provides
managed security monitoring services to organisat ions
w orldw ide. He also designed the popular Blow f ish
encrypt ion algorithm, and his Tw of ish w as a f inalist for
the new US Federal Advanced Encrypt ion Standard
(AES).
Q. Are the commercial and economic threats f rom
malicious code at tacks greater than f rom plain buggy
sof tw are?
A. They're the same. Buggy software contains vulnerabilities;
malicious code is created to exploit those vulnerabilities. There
are additional non-security-related vulnerabilities from buggy
code-crashes and data loss-but the malicious attacks are
directed and nasty.
Q. Formal proof techniques for sof tw are have been
around a long t ime. Why don't commercial sof tw are
developers use them more?
A. Because they don't work for anything larger than toy
programs. They're in the early theory stage, and don't really
have any practical applications in modern software
development.
Q. Would greater use of such techniques offer more
protect ion against malicious at tacks?
A. I doubt it. My guess is that they would offer less protection,
because users would have a false sense of security.
Q. What advice w ould you give IT directors about how
to improve the reliability of their (commercial) systems?
A. Defence in depth. All security fails eventually. All products
have vulnerabilities. You need defence in depth.
The law of diminishing returnsprompts software house torelease programs, knowingthey have bugs.
Schneier: all security fails eventually
means buying new hardware, or at least returning the system to the
manufacturer to be upgraded. But advances in solid state memory
technology and faster fixed and wireless communications speeds
mean that it is more and more practical to download new software
to hand-held devices.
But that flexibility also opens the door to rogue influences.
Basically one trades flexibility for performance and security. CE
Shannon, writing in 1948, described the problem of software
reliability thus: "The system must be designed to operate for each
possible selection, not just the one which will actually be chosen,
since this is unknown at the time of design."
Clearly, reducing the number of things that can go wrong helps,
but that also means that the system can't do very much. Andy Gray,
chief technology officer of Icons, a computer security consultancy,
comments "In today's environment, where functionality and
release schedules govern software development, security takes a
back seat."
Cl ick w rapping aw ay l iab i l i t y
Small wonder then that software suppliers have fought so hard to
avoid liability for their products. In 1998 they won US legal
recognition for so-called "click wrap" user agreements. These
absolve them from damages from any failure of the software to
perform as intended for any reason, no matter what the
consequences.
But this may be less of a problem than one thinks. Cambridge's
Anderson points out that most software bugs don't threaten
system security. But he also notes that it is virtually impossible to
eliminate all bugs. Put another way, killing the last 10% of the bugs
take 90% of the time and the cost.
Indeed, it is the law of diminishing returns that in many
situations prompts software house to release programs into the
"wild", knowing they have bugs, and to incorporate facilities
aimed at limiting the effects of residual bugs. Not only would it
cost too much and take too long to deliver bug-free software, but
their testers cannot hope to cover all the scenarios under which
users will try to use the software. It is therefore economically
rational to release buggy software and to correct the worst bugs as
they are reported.
This thinking underpins the open access movement that has led
to the development of GNU/Linux. Developing software in the
open means that anyone with an interest and/or enough ego can
have a go at improving it. Goodness (as in killing bugs) may be its
own reward, but the enhancement of one's reputation may also
lead to better jobs. Moreover, many major IT companies, not least
IBM, are now contributing to the open software movement.
Although Microsoft preaches the benefits of closed system
development, it practises an approximation to the opposite. In
Windows XP it explicitly acknowledges this by having the system
prompt the user to go on-line to report some error conditions
when they arise.
M icrosoft has certainly benefited from its customers'
willingness to put up with buggy software. But its past reluctance
to do much about it prompted the rest of the IT industry to get
behind Linux.
This threat to its market share has forced Microsoft to react.
Over the past couple of years it has sought to push its concept
of Trustworthy Computing. Introducing the initiative in July 2002,
Microsoft boss Bill Gates said "Technology … is far too important
to let a few criminals stop the rest of us from enjoying its amazing
benefits." In February 2003 he appointed a panel of 19 experts to
advise it on "security, privacy and reliability enhancements in
Microsoft products and technologies".
At the time Scott Charney, M icrosoft's chief security strategist,
said "Achieving trustworthy computing will take many years and
require thoughtful and sustained collaboration between the
industry and academic communities." Board member Neeraj Suri,
professor at Technische Universität Darmstadt, added "Trust in the
e-world is not an option anymore. It behooves us socially,
economically and scientifically to ensure that trust in a system
becomes a foundational premise."
However, Cambridge's Anderson believes that M icrosoft's
Trustworthy Computing is anything but. "TC will enable
Microsoft to lock in its customers more tightly, so it can charge you
more. The proposed mechanisms could also have some disturbing
consequences for privacy, censorship, and innovation," he says.
Ian Grant is a business writer who specialises in IT and
innovation issues.
te
ch
no
lo
gy
41
Info
secu
rity To
day
May/Ju
ne 2
004
The idea is to catch bugsearlier when they are cheaper
to correct.
