Many software security vulnerabilities occur because of poor programming practices. Vulnerabilities are often algorithmically detectable by static source-code scanners designed for identifying potential security issues. As the number and severity of potential security holes per line of code increase, we believe that the overall quality of the source code in terms of security decreases. BogoSec metrics are computed values that attempt to reflect relative ratings of source code security quality for comparative purposes.
The motivation behind BogoSec is to influence developers to produce more secure source code over time. The various scanners that exist point developers to potentially insecure sections of code. Developers are often reluctant to use such scanners because of a seemingly high degree of "false positive" output as well as the difficulties associated with use. BogoSec attempts to reduce the penalty of false positives while broadening the scope of the source scan by utilizing multiple independent scanners. This produces high level metrics allowing developers and users alike to comparatively judge the quality of the source code in terms of security.
Several source-code scanners exist that identify numerous vulnerabilities with varying accuracy and success. BogoSec parses the output of any number of source-code scanners and computes its metric based on the number, severity, and frequency of potential bugs found as per number of lines scanned. BogoSec currently supports the following scanners:
Support for additional scanners is easily extended by creating plugins that parse the output of the new scanners in the same way that existing plugins parse the currently supported scanners. This is useful for incorporating support for proprietary or internal scanning tools.
BogoSec requires that at least one of the above scanners is installed on the system and can be found in the path. These scanners are not distributed as part of BogoSec. However, BogoSec does include plugins that interface with each scanner.
The basic methodology of BogoSec is as follows:
BogoSec Final Score |
= | total vulnerability severity points from all scanners total lines of code analyzed by all scanners |
The algorithm above considers weighted vulnerabilities. The total number of vulnerability severity points accumulates as scanners identify potential vulnerabilities. The ratio of points per line is the indicator by which users of BogoSec are able to compare source code in terms of security quality. BogoSec operates under the assumption that as the number of weighted severity points per line increases, the overall security of the compiled code decreases.
BogoSec is implemented as a Perl script and accompanying Perl modules. One can add support for additional scanners by creating a module that executes the scanner against the source code, interprets the output, and normalizes severity ratings to BogoSec's scale.
Initially, BogoSec sets up its execution environment by parsing configuration files and reading command line parameters. (See the manpage for an extensive description of command line options.) The final parameter specifies the target, which may be a single source file, an entire source tree, an archive {*.tar.gz, *.tgz}, or a source RPM {*.src.rpm}. BogoSec handles the last two formats by creating a temporary directory and expanding the code accordingly.
The script builds a list of files in the target tree, consisting of any case insensitive file matching {*.c, *.h, *.cpp, *.c++}. Currently, scanners are executed on one file at a time. Although some scanners have the ability to recursively scan an entire tree, the stability and consistency of BogoSec favors compiling the results of individually scanned files. BogoSec is multithreaded such that all three scanners execute simultaneously on the list of files, waiting for the slowest scanner to complete execution before analyzing the results.
Each scanner's plugin has a routine defined for analyzing the particular output of that scanner. This consists of scanning each line of output to determine if the line specifies an identified potential vulnerability. If it does, the filename, line number, severity, and description are parsed out according to rules defining each scanner's output conventions. The severity is scaled according to BogoSec's scale (by default, a 10-point scale). The plugin obtains the number of lines of source code scanned from the output whenever possible. This data is stored in a structure accessible by the main program.
Finally, the main program tallies the number of points accumulated by all scanners on all files, as well as the number of lines of code scanned by all scanners in all files. The BogoSec final score is this quotient, which the script reports and then exits.
BogoSec is installed through the bundled makefile, using the "make install
" target. The installation creates the following:
/usr/local/bin/bogosec
/usr/local/bin/bogosec_wrapper
/etc/bogosec.conf
/usr/local/bogosec/
/usr/local/bogosec/plugins/
/usr/local/bogosec/documents/
/usr/local/bogosec/plugins/Bogo{Flaw,ITS4,Rats}.pm
BogoSec is executed as any other command-line utility. Arguments are paths to the source code to be evaulated.
Example:
# bogosec tests/apache/apache_1.3.31.tar.gz Running flawfinder... Running rats... Running its4... flawfinder 5212 points 75682 lines its4 9502 points 110022 lines rats 8573 points 99234 lines >>> Using scanners: (flawfinder its4 rats ) >>> 23287 total severity points >>> 284938 total lines of code scanned >>> final score = 0.0817277208843072
In the above example, BogoSec executed against the Apache source tree archived in tests/apache/apache_1.3.31.tar.gz
.
BogoSec can also evaluate individual files or uncompressed source
trees. For a discussion of the various options associated with BogoSec,
please see the manpage.
To test the operation of BogoSec and the reliability of its results, test cases have been executed, documented, and studied. Tests were executed against several popular packages against all released versions available for download. These tests demonstrate BogoSec's use against a given package to indicate the general trend of the quality of source code. Also, several equivalent open source software packages (when available) were tested to compare in an absolute sense which of the packages have better BogoSec quality scores.
Apache is the world's most popular Web server, accounting for over 60% of all internet Web sites. The Apache 1.3 tree has been under constant development since 1998 and up to the present. In that time, the Apache team has added many features and fixed many bugs. Apache 2.0 is the next generation Web server from the Apache development team.
These test results show consecutive runs of BogoSec against all Apache 1.3 (in maroon) and 2.0 (in blue) released verisions available.
Absolute Points: The overall scores of 1.3 are very slowly but smoothly increasing. On the other hand, the tremendous spike between 2.0.18 and 2.0.28 (both beta releases) should cause some concern. Also, it looks like the 2.0.44 release fixed some security problems present in previous releases.
Points / Line: Both versions exhibit BogoSec ratios that are generally improving over time, though the 2.0 releases have better scores than the 1.3 branch.
Absolute Points | Points / Line |
---|---|
![]() | ![]() |
(continued) Package Version Score LOC Points Package Version Score LOC Points apache 1.3.0 0.0896 186578 16711 httpd 2_0_15 0.0537 409238 21965 apache 1.3.11 0.0811 256983 20841 httpd 2_0_16 0.0573 413056 23673 apache 1.3.12 0.0812 257446 20895 httpd 2_0_18 0.0555 468691 26035 apache 1.3.14 0.0809 267005 21595 httpd 2_0_28 0.0686 570302 39112 apache 1.3.17 0.0809 270501 21889 httpd 2.0.32 0.0676 594210 40162 apache 1.3.19 0.0799 271083 21654 httpd 2.0.35 0.0672 617261 41503 apache 1.3.1 0.0879 190542 16753 httpd 2.0.36 0.0665 624265 41540 apache 1.3.20 0.0797 272119 21681 httpd 2.0.39 0.0651 639129 41620 apache 1.3.22 0.0814 276845 22546 httpd 2.0.40 0.0659 648812 42737 apache 1.3.23 0.0808 280312 22660 httpd 2.0.42 0.0649 661539 42933 apache 1.3.24 0.0798 285126 22761 httpd 2.0.43 0.0646 664596 42925 apache 1.3.27 0.0796 287594 22900 httpd 2.0.44 0.0571 678224 38701 apache 1.3.28 0.0790 290025 22923 httpd 2.0.45 0.0568 685602 38918 apache 1.3.29 0.0791 290240 22965 httpd 2.0.46 0.0569 688417 39160 apache 1.3.2 0.0888 195124 17329 httpd 2.0.47 0.0566 689951 39044 apache 1.3.31 0.0817 284938 23287 httpd 2.0.48 0.0565 693187 39191 apache 1.3.32 0.0816 285458 23305 httpd 2.0.49 0.0603 655544 39517 apache 1.3.33 0.0816 285501 23305 httpd 2.0.50 0.0597 658322 39305 apache 1.3.3 0.0889 196542 17479 httpd 2.0.51 0.0594 661497 39268 apache 1.3.4 0.0871 202112 17596 httpd 2.0.52 0.0594 661740 39308 apache 1.3.6 0.0880 206900 18209 apache 1.3.9 0.0806 247724 19963 |
OpenSSH provides an encrypted command shell, usually for remote network access to systems. OpenSSH is primarily developed by members of the OpenBSD project, a group of developers known for security-conscious code. OpenBSD conducts extensive manual audits of source code in order to identify and fix security vulnerabilities.
These graphs demonstrate admirable models for secure software development. Both sets of data seem to approach asymptotes, with Absolute Points gradually increasing, and Points/Line scores gradually decreasing, and neither have significant spikes. The false positives reported by the tools probably form the asymptote base of these graphs.
Absolute Points | Points / Line |
---|---|
![]() | ![]() |
(continued) Package Version Score LOC Points Package Version Score LOC Points openssh 2.1.1p4 0.1329 92619 12307 openssh 3.2.2p1 0.0961 159646 15342 openssh 2.2.0p1 0.1305 93999 12267 openssh 3.2.3p1 0.0961 159660 15342 openssh 2.3.0p1 0.1252 108205 13549 openssh 3.3p1 0.0954 161720 15430 openssh 2.5.1p1 0.1128 122159 13778 openssh 3.4p1 0.0952 161964 15416 openssh 2.5.1p2 0.1126 122387 13778 openssh 3.5p1 0.0953 166756 15899 openssh 2.5.2p1 0.1106 128989 14266 openssh 3.6.1p1 0.0945 170301 16101 openssh 2.5.2p2 0.1106 129019 14266 openssh 3.6.1p2 0.0945 170316 16101 openssh 2.9.9p1 0.1040 143107 14880 openssh 3.6p1 0.0946 170270 16101 openssh 2.9.9p2 0.1040 143107 14880 openssh 3.7.1p1 0.0895 178101 15943 openssh 2.9p1 0.1048 133867 14029 openssh 3.7.1p2 0.0894 178288 15943 openssh 2.9p2 0.1048 133972 14040 openssh 3.7p1 0.0895 178084 15943 openssh 3.0.1p1 0.1039 143342 14899 openssh 3.8.1p1 0.0864 181919 15722 openssh 3.0.2p1 0.1039 143351 14899 openssh 3.8p1 0.0895 181852 16269 openssh 3.0p1 0.1039 143319 14897 openssh 3.9p1 0.0869 186467 16203 openssh 3.1p1 0.1018 144189 14683 |
Vsftpd and wu-ftpd are two major open source FTP servers. The first, vsftpd, was written with security as a primary objective, with the "vs" standing for "very secure". The second, wu-ftpd, is Washington University's FTP server, which pre-dates vsftpd and has had a long and colorful history of some high profile security vulnerabilities.
These charts show a drastic difference in source-code security between vsftpd and wu-ftpd. Both BogoSec scores of vsftpd appear orders of magnitude better than wu-ftpd. This is consistent with the popular opinion regarding the security of these two FTP servers.
Absolute Points | Points / Line |
---|---|
![]() | ![]() |
(continued) Package Version Score LOC Points Package Version Score LOC Points vsftpd 0.9.2 0.0201 25656 515 wu-ftpd 2.0 0.2912 24345 7089 vsftpd 1.0.1 0.0197 26149 515 wu-ftpd 2.1 0.3282 24334 7986 vsftpd 1.1.0 0.0196 27784 544 wu-ftpd 2.2 0.3403 26388 8980 vsftpd 1.1.1 0.0194 28029 544 wu-ftpd 2.3 0.3403 26385 8980 vsftpd 1.1.2 0.0188 28868 544 wu-ftpd 2.4.2 0.2743 35509 9741 vsftpd 1.1.3 0.0194 29228 568 wu-ftpd 2.4 0.3384 26495 8966 vsftpd 1.2.0 0.0194 32849 637 wu-ftpd 2.5.0 0.2754 54752 15081 vsftpd 1.2.1 0.0185 34409 637 wu-ftpd 2.6.0 0.3093 63603 19670 vsftpd 1.2.2 0.0185 34540 637 wu-ftpd 2.6.1 0.3047 65322 19903 vsftpd 2.0.0 0.0195 37607 734 wu-ftpd 2.6.2 0.3051 65212 19898 vsftpd 2.0.1 0.0195 37657 734 vsftpd 2.0.2 0.0194 37883 734 |
Postfix, qmail, and sendmail are major open source mail transfer agents (MTAs). Sendmail has long served as the primary mailer for UNIX® environments. But sendmail's history has been riddled with exploitable security vulnerabilities. Recently, several alternative MTA's have emerged, such as postfix and qmail. Postfix was written and is maintained by an expert member of IBM Research in secure computing. Qmail is another option with a notably small code base designed in the interest of security.
Absolute Points | Points / Line |
---|---|
![]() | ![]() |
(continued) Package Version Score LOC Points Package Version Score LOC Points postfix 1.0.8 0.0241 199281 4800 sendmail 8.11.7 0.0894 232285 20775 postfix 1.1.13 0.0222 237491 5273 sendmail 8.12.10 0.0501 324450 16251 postfix 2.1.5 0.0193 293908 5683 sendmail 8.12.11 0.0500 325152 16265 qmail 0.92 0.0669 44957 3008 sendmail 8.13.0 0.0494 330588 16330 qmail 1.00 0.0660 44913 2966 sendmail 8.13.1 0.0493 331157 16341 qmail 1.01 0.0679 44918 3050 sendmail 8.13.2 0.0493 331607 16335 qmail 1.02 0.0693 45339 3143 sendmail 8.13.3 0.0493 331647 16335 qmail 1.03 0.0722 46737 3377 |
Open source scripting languages include perl, php, python, and ruby. Each of these provide a higher level programming language easily used by developers to very easily and quickly accomplish software tasks. A binary interpreter is needed by each of these languages to executed the scripted source code.
Ruby and Python have the lowest absolute and ratio scores. PHP, by far, has the largest code base and the highest absolute points. In all cases, the latest release shows marked improvement over the earliest release.*
Absolute Points | Points / Line |
---|---|
![]() | ![]() |
(continued) Package Version Score LOC Points Package Version Score LOC Points perl 5.005 0.0502 278627 13982 Python 2.1.3 0.0315 628654 19797 perl 5.6.1 0.0456 386938 17649 Python 2.2.3 0.0247 746199 18461 perl 5.8.6 0.0457 523662 23908 Python 2.3.4 0.0263 895668 23539 perl 5.9.0 0.0475 500034 23765 Python 2.4 0.0214 980119 20976 perl 5.9.1 0.0484 498403 24145 ruby 1.3.5 0.0580 203286 11792 php 3.0.18 0.0753 406914 30655 ruby 1.3.7 0.0578 203739 11781 php 4.2.0 0.0512 876430 44854 ruby 1.3 0.0539 183631 9893 php 4.2.3 0.0516 893882 46141 ruby 1.4.0 0.0578 204141 11799 php 4.3.10 0.0441 1227476 54180 ruby 1.4.6 0.0572 207843 11887 php 4.3.8 0.0446 1216570 54224 ruby 1.6.0 0.0561 219104 12299 php 4.3.9 0.0444 1221208 54194 ruby 1.6.8 0.0510 240446 12267 php 5.0.0 0.0361 1433945 51758 ruby 1.8.0 0.0421 396570 16687 php 5.0.1 0.0358 1431329 51212 ruby 1.8.1 0.0394 426031 16778 php 5.0.3 0.0357 1441051 51379 ruby 1.8.2 0.0353 492828 17405 |
The following data shows an absolute comparison of the latest release of all packages tested above. One must be somewhat cautious of this comparison, because it's very difficult to compare vastly different packages especially when one package consists of a few hundred lines of code and another comprises millions of lines of code. Still, this chart teaches some valuable lessons about the advantages of BogoSec as well as perhaps some of its shortcomings.
Absolute Points: It seems that vsftpd, qmail, and postfix are clear winners--these being packages designed and implemented by experts to be secure software. On the other end of the spectrum, httpd and php totaled the highest number of points--probably because of the fact that these are two of the largest packages that were tested.
Points / Line: Again, postfix and vsftpd exhibited the best scores, while wu-ftpd scored several orders of magnitude worse than the rest of the field--consistent with the popular opinion of its state of security.
All four of the scripting languages (python, php, ruby, perl) appear to have relatively similar BogoSec scores. It is somewhat surprising to see apache, httpd, and openssh
near the upper end of the spectrum. This also deserves further
investigation as it is possible that these packages are yielding an
abnormally high number of false positives, thereby driving their scores
disproportionately higher.
Absolute Points | Points / Line |
---|---|
![]() | ![]() |
Package Version Score LOC Points postfix 2.1.5 0.0193 293908 5683 vsftpd 2.0.1 0.0195 37657 734 Python 2.4 0.0214 980119 20976 ruby 1.8.2 0.0353 492828 17405 php 5.0.3 0.0357 1441051 51379 perl 5.9.1 0.0484 498403 24145 sendmail 8.13.3 0.0493 331647 16335 httpd 2.0.52 0.0594 661740 39308 Qmail 1.03 0.0722 46737 3377 apache 1.3.9 0.0806 247724 19963 openssh 3.9p1 0.0869 186467 16203 wu-ftpd 2.6.2 0.3051 65212 19898 |
We hope that BogoSec drives developer awareness of insecure code by providing a higher level interface to numerous scanners. One way of encouraging developers to use these existing tools is by integrating BogoSec into some common development tools and processes, such as:
To demonstrate our point, we have created wrapper scripts that integrate BogoSec into the CVS source code repository on both the client and server side. While these example scripts use statically configured thresholds, it is conceivable to extend the scripts to check the score of previous commits and only allow new code that is at least as good as what is currently in the repository. In this way, one might try to ensure that the security quality of some code base never decreases.
We have also created a wrapper script that invokes the tool on a directory which may contain any number of file formats supported by BogoSec. The script will run BogoSec on each file and output the results in a tabulated format.
Proactive developers can make use of the script "client_cvs_wrapper
" to be used in place of the "cvs commit
" command. Defined in this script are two thresholds, MAX_SCORE
and MAX_SEV_PTS
. The BogoSec scores of the code to be committed must meet both of these requirements, or else the developer is notified to fix some of the suspected vulnerabilities. The user can configure the script to either entirely block the commit or simply issue a warning. Developers can use this script without bothering the server administrator.
Heavy-handed administrators can set MAX_SCORE
and MAX_SEV_PTS
thresholds in a script "server_cvs_wrapper
", which is called by "CVSROOT/commitinfo
". The wrapper informs the user of the BogoSec scores and exits with a non-zero return code when the code's scores do not meet requirements, thus preventing the code commit (or optionally issuing a warning).
This is an automated script that administrators may use to run BogoSec on a directory of different file formats to help them identify potential weak points of the system. The script will collect the results of the multiple BogoSec executions and arrange them in a concise and easy to read tabulated format { Package, Sev Points, Lines of Code, Final Score}.
BOGOSEC(1) BogoSec User Manual BOGOSEC(1) NAME bogosec - source code quality metric using established static source code scanners SYNOPSIS bogosec [-l] [--log-dir directory ] [--min-sev 0-10 ] [--nhf] [-p plu- gin_name [args] ] [--plugin-dir directory ] [--sev-range-max num ] [--temp-log-dir directory ] [-v 0|1 ] [--xp plugin_name ] TARGET bogosec_wrapper TARGET-DIRECTORY DESCRIPTION BogoSec attempts to influence developers to produce more secure source code over time. Various existing scanners point developers to poten- tially insecure sections of code. BogoSec broadens the scope of source code scans by utilizing multiple independent scanners and compiling the results into high-level calculated metrics. These metrics can help developers and users alike to comparatively judge the security quality of source code. bogosec_wrapper automates the process by running bogosec on a direcotry containing different file formats supported by bogosec and collecting the results in : /tmp/bogosec.results and /tmp/bogosec.detail_results (overwritten by each invocation). Keep in mind, bogosec_wrapper does not accept any options. OPTIONS -l Turn on scanner output logging. Log will be called <scan- ner_name>.log and created in current working directory, unless --log-dir is used to specify a different location. --log-dir directory Specify a directory for scanner output logs (only makes sense if -l is also used). Default is current working directory. --min-sev minimum_severity_level Specify a minimum severity level. Any vulnerabilities reported by the scanners whose score falls below this number will be ignored. The argument must be a number 0-10. Default is 0. --nhf, --no-header-files Do not scan header files. Useful if the scanners being used do not support scanning header files. -p, --plugin plugin_name [args] Specify a plugin to use. If no plugins are defined on the command line, all of the plugins in the plugins_dir will be used. This option can be passed more than once to specify a set of scanners to use. Each scanner requires a separate instance of the --plugin flag (please see examples). Optionally, a set of command line arguments can be passed to the scanner - this fea- ture must be used with care. Keep in mind that the plugin requires a certain formatting of the scanner output (for exam- ple, '-SQ' is always passed to flawfinder, and '-w 3' is always passed to rats). You can pass additional command line arguments using this option, but be aware of the effect it might have on the formatting of the scanner output, and the effect that will have on the plugin's ability to parse it correctly. If you must change the defaults ('-SQ', '-w 3', etc.) you must edit the plu- gin directly. --plugin-dir directory Specify the directory where the plugins are stored. Default is /usr/local/bogosec/plugins. --sev-range-max number Specify the maximum severity value to be used in calculating the severity value range. The default is 10. For example, setting --sev-range-max to 50 would mean that the severity results would now be on a scale of 0-50 instead of on a scale of 0-10. This can be used to scale the result if more granularity is required. NOTE: -v 1 will not work if this option is used. --temp-log-dir directory Specify a directory where you want the temporary files used by BogoSec to be stored (scanner output logs, etc.) The default is /tmp/. -v, --verbosity 0|1 Specify verbosity level (default is 0). If 1, then a graph of the severity points is shown, which breaks the results down by severity levels. This option does not work if the --sev-range- max is changed from 10. --xp, --exclude-plugin plugin_name Do not run plugin defined by plugin_name. FILES /etc/bogosec.conf Global configuration file. The settings here are overwritten by any settings in user's ~/.bogosecrc file. ~/.bogosecrc Default user configuration file (overrides the settings in /etc/bogosec.conf). This file is not created during an instal- lation, you must create it yourself. /usr/local/bogosec/plugins/ Default plugins directory. Can be changed with --plugin-dir option. Plugins must be executable, and must end in .pm as per convention. /usr/local/bogosec/documents/ Directory of BogoSec documentation and other germane documents. BUGS Not all input validated. Not all environmental variables checked. This program expects to be run by trusted users. AUTHORS Developed by Dustin Kirkland, Agoston Petz, and Loulwa Salem at the IBM Linux Technology Center. Linux Jan 25 2004 BOGOSEC(1)