Free software (open source or not) is interesting for many reasons. It can be adapted to your own needs, it can be easily integrated within complex architectures but the most important remains, of course, the price. Even if they are many hidden costs related to "free" software. In case of issues, a lot of time may be spent in searching for a solution or diving into the source code (and everybody knows that time is money!).

Today, more and more organisations are not afraid anymore to deploy free software in their infrastructure but are those solutions really secure? A customer came to me with an interesting question about performing a security audit of free software. The idea is to validate the software before deploying it in his (very sensitive) infrastructure. This could be seen as a kind of quality insurance.

The idea is not to perform a deep source code review or to pentest the tool but more to establish a checklist of key points. I already compiled a rough list of questions that I'd like to share with you:

• What is the programming language used?
  Not all languages have the same security maturity. Some applications rely on functions that may become obsolete and prevent regular updates (PHP is a good example). Is a specific version of the language required? (eg. Python 2 VS. Python 3). Also writing secure code in C or Perl isn't as easy as other languages.
   
• Architecture and security framework?
  Can the application be deployed in a virtual environment, is there a Docker container available? Are they other software components required (like a database server, a Redis server, ...). How easy may the application be deployed in high-availability? Does it work properly behind a reverse-proxy (in case of a web application)? Is it integrated or another software layer is required?
   
• Regular updates?
  Pieces of software have bugs and security issues. How are patches released and how often? How reactive are the developers? If there is a bug tracking system, check the average time required to fix an issue. Some projects are very impressive and roll out fixes within hours of being notified while others aren't as impressive. It must meet the same patching requirements as all other software.
   
• Roadmap for the coming months?
  Is there a roadmap in place with new features? Can users propose and submit new features?
   
• How big is the community around the project?
  Check out forums, mailing lists, issues reported on github.com. Are they regular commits? Are developers easily reachable? How many messages are posted on public resources?
   
• How big is the current users base?
  If the existing users base is large, there are chances that bugs will be found quickly and new features suggested. Do current users look happy while using the application?
   
• The documentation and quality of documentation within the code?
  A good documentation of the code means that it is maintainable. Documentation is mandatory to operate the application and to help to resolve issues.
   
• Are external pieces of code (like libraries) used?
  Most applications rely on external libraries (think about OpenSSL). These pieces of code must follow the same process.
   
• Data management?
  How safe is data used / generated by the application? Think about data "in use", "at rest" and "in move". Are backup and restore procedures available?
   
• Is there enough granularity in the configuration?
  How the application supports complex configuration? How deep can it be customized? Does it implement AAA (Authentication, Authorization & Accounting), RBAC (Role Based Access Control). 
   
• Extendability of the application?
  Is the application modular and can be expanded with modules or plugins? If you consider a popular application like Wordpress, the core is fairly well maintained, but plugins introduce most vulnerabilities, and may or may not get maintained very well, so you might suggest that 3rd party modules added to a free package be treated as separate pieces of software.
   
• Age and maturity of the project?
  If a new software has been released yesterday, it will be more difficult to review it.
   
• What is the license used to distribute the software?
  The license must be reviewed to ensure there are no restrictions on its intended use. A lot of free software is not free for corporate use. If you have plans to adapt the software to your own needs, how to not break the existing license?
   
• Integration with other tools?
  Is is easy to integrate the applications with authentication mechanisms (LDAP, RADIUS)? Can logs be easily exported? (Syslog, CEF)
   
• Are they CVE related to the software?
  How many CVE's released in the last x months? What is the severity of CVE's? Repeated issues in CVE's (did they just fix that one instance or look at the rest of the code base for similar issues and fix them as well). If a piece of software released recently has no CVEs associated with it, it doesn't indicate that it's secure. More CVE’s could mean that code quality is not good or that more people are analyzing  the code and finding holes. On the other side, a  lack of CVE’s doesn’t necessarily make a particular piece of code more secure. I suggest keeping track of CVE’s issued and how quickly patches are released to correct CVE’s and basing your decision on that. 
   
• Used in commercial solutions?
  If commercial companies embedded the project in their products, these companies have huge teams of developers that could ensure that security issues will be fixed.
   
• Commercial support?
  Can a freelance be hired to help in the integration or the debug of issues?

As a final remark, there is no specific reason not to perform the same for commercial products. Most commercial products either use or depend on open source libraries/software. Also keep in mind that a commercial product doesn't mean a secure one.

Do you have other controls that could be added to the list? Feel free to share your comments!

Xavier Mertens (@xme)
ISC Handler - Freelance Security Consultant
PGP Key