Detection, Prevention and Analysis of Bugs in OpenBSD

Note: This project was accomplished as a 2005-2006 Final Year Project by Niall O'Higgins (report).


Summary of Niall's Work

Detection, Prevention and Analysis of Bugs in OpenBSD

Winner of IBM Open Source Software competition
Niall O'Higgins, UCD 2006

OpenBSD is widely regarded as the world's most secure operating system. What is it that places it so far ahead of even commercial operating systems whose vendors invest millions in security? The core OpenBSD philosophy is that security holes are the result of programmer error, that they are another class of bug. Thus, the best way to avoid security holes is to write correct code. In other words, security is a side-effect of software correctness.

The OpenBSD source code has been and continues to be relentlessly trawled for bugs. The cause of a bug could be for example due to a misunderstanding of an API, an incorrect assumption on the part of the programmer, or a simple typo. From time to time, a bug is discovered which really represents a whole new class of mistake. Once such a class is discovered, the entire source tree is scoured for further instances of this error - thus one bug fix may lead to many more, often solving problems in other unrelated places before they are even noticed.

A large number of these bugs have been located solely by human auditing of the source code. While remarkably successful, there must exist automatic approaches to finding software bugs. One method developed by the author specifically targets UNIX command-line applications. Such applications can be viewed as functions which transform some input, typically files, standard input and command-line arguments, into some output, typically some files, exit codes and standard output. These inputs almost always have interesting boundary points, for example a numeric input may be unable to handle huge negative numbers or a string input may be unable to handle random binary data. It is possible to automatically extrapolate these boundary points and generate large numbers of combinations of input data, testing interesting runtime cases far more quickly than any human.

A relatively crude proof-of-concept implementation was successfully employed during the course of this project, revealing many bugs in portions of the OpenBSD source tree. The results showed that this form of automatic application testing, while imperfect, is nonetheless very useful in practise for examining software correctness.


Original Project Specification

  • Supervisor Dr. Joseph Kiniry
  • Subject Area Software Engineering in Operating Systems
  • Pre-requisite Very good knowledge of C and computer architecture
  • Co-requisite (things you must learn along the way) Good knowledge of UNIX (generally) and OpenBSD (specifically), OS-level systems programming, Databases, Web CGI programming
  • Subject Coverage Programming Languages, Compilers, Information Systems, Static Analysis
  • Project Type Analysis, design, and implementation
  • Hardware/Software: Any machine running OpenBSD

Description

OpenBSD is widely regarded as the world's most secure operating system. It represents the combined effort of hundreds of dedicated individuals, most of whom work in their spare time for free, over nearly a decade.

OpenBSD developers try very hard to write code that is correct, a goal which has the convenient side effects of producing code which is also well-designed, maintainable, clean and secure.

The OpenBSD source code has been and continues to be relentlessly trawled for bugs. The cause of a bug could be for example due to a misunderstanding of an API, an incorrect assumption on the part of the programmer, or a simple typo. From time to time, a bug is discovered which really represents a whole new class of mistake. Once such a class is discovered, the entire source tree is scoured for further instances of this error - thus one bug fix may lead to many more, often solving problems in other unrelated places before they are even noticed.

Finding, analysing and fixing bugs has been central to the success of the OpenBSD project. However, no formal database is maintained of the bugs encountered nor the tools and methodologies which help in the process of indentfying and fixing them.

This project focuses on (a) identifying and fixing bugs in the OpenBSD source code (b) analysing and classifying these bugs, with resulting information made available in a searchable Internet database (c) exploring the development of software engineering tools and processes to aid in this endeavour.

Mandatory

  1. Evolve a structured and systematic methodology to find bugs in the OpenBSD source tree.
  2. Design and implement a flexible Internet-accessible database for the storage of bug-data.
  3. Develop a concise process for analysing and categorising bugs.
  4. Find, fix, analyse and categorise a quantity of bugs in the OpenBSD source tree.
  5. Write a paper on the results of this analysis.

Discretionary

  1. Propose guidlines for both the API designer, API implementor and API user to avoid pitfalls in the future.
  2. Design software tools to aid in the discovery of bug instances.

Exceptional

  1. Implement software tools which automatically locate instances of a bug class in a source tree.

Sources of information and preparatory reading

  1. The OpenBSD web site