Penetration Testing – Who Should Be Doing This Anyway?

If you are responsible for the security of a product in development, you may be wondering what the options there are for security/penetration testing. First, you may need to know what is involved with security testing and more specifically, what is a penetration test? A penetration test, or pentest, is a method of assessing the security of a computer product, system or network by simulating an attack from a malicious source, which can be defined from a threat model. In most cases, a pentest is performed to validate security before shipment of the product with the overall purpose of providing assurance that the security objectives and mitigations are correctly implemented, thus limiting the potential risk of exposure.

One important concept that is needed for anyone responsible for deciding who performs a penetration test is the difference between white box, black box, and grey box penetration testing. The difference is in the amount of knowledge of the infrastructure or product to be tested.

  • White Box Testing – provides the testers with complete knowledge of the product or infrastructure to be tested. Often information provided to the testers includes architectural specifications, source code, infrastructure information including network diagrams and IP addressing information.
  • Black Box Testing - assumes no prior knowledge of the product or infrastructure to be tested. The testers must try to figure out the inner workings of the product or infrastructure based on analysis of packaged documentation, shipped assemblies, inputs and outputs.
  • Grey Box Testing - There are also several variations in between, often known as grey box tests. Penetration tests may also be described as "full disclosure", "partial disclosure" or "blind" tests based on the amount of information provided to the testing party.

So, who should perform a Penetration Test?

Many organizations are training software developers and architects to improve secure development practices. Some product development teams have begun using static source code analysis tools as a means to determine insecure function calls, nonexistent or improper input validation and less secure coding practices. By all means, the increased security knowledge within a development team will definitely improve security as it is integrated into the Product Development Life Cycle (SDLC) and help in the awareness of potential security issues during requirements gathering, architecting, designing and coding of a product. But in many cases, the investment in security training and tools for developers is expected to pay off in that penetration testing before shipment might be deemed unnecessary. Although, these examples of best practices can really improve the security in development, it should not nullify the need of formal security testing that is provided in a penetration test.

It is also important to note that vulnerabilities found in any stage of development or penetration test may not have a known exploit. This means that a buffer overrun discovered during any security testing should not have to be exploited to prove that it is not secure. Writing exploits to prove code is insecure may be out of scope for a security test due to the time it can consume. Any potential security issue identified during a Penetration Test should be evaluated based on risk and if necessary, mitigated using secure coding practices.

Internal testing within the product development team:

Pros - An advantage to this type of testing is that some security related issues can show up early on in the development cycle, allowing adequate time for changes to be made at the architectural or design phases. The developers can perform the security related tests with the white box approach as they would have the most complete knowledge of the product. This can also be beneficial if there are talented individuals within the groups that can share their knowledge, it could be a great benefit to others in the groups. Additionally, if security related tools have been purchased by the corporation, the use of these tools can be leveraged.

Cons - There may be a lack of experience for the tester. If an attempt is made to find an experience security tester, the resource may not be available. Another likely con for developers testing security in their own code is that the mitigations or security objectives might not be tested as affectively due to a lack of objectivity which is important for any testing of a product. Quality Assurance (QA) team could get involved but it is important to note that any QA testing team is responsible for ensuring functionality in the deliverables and are not usually specialized in this area.

Insource - Testing within the same company using an internal dedicated security validation team:

An internal team may be already in place in some organizations and can be engaged to completely manage the security testing for a product. There are positives and negatives to this strategy as well.

Pros – This option can allow for sharing of knowledge of the product with the security testers and developers who can communicate directly with each other and discuss architecture, design and implementation. The employees may be permanent employees that allows for less legal agreements than a completely outsources solution when there are IP concerns. This can provide the ability for full white box testing if desired. The sharing of security testing knowledge is more possible and can achieve a higher level of objectivity in testing than in the case of a development group performing the security tests. The biggest benefit to this option is that it can allow the most open and direct communication and collaboration between the security testing team and the developers. The Security validation team can use a white box approach as they would have access to the complete knowledge of the product by communicating with the development team.

Cons – Expert security testers are limited resources these days so it may be necessary to pull in less experienced resources who can request support from those who are more experienced. If a formal certification is required for the product, this option may not be good as the organization would be self-certifying its security in a product.

Outsource - Security Penetration Testing using an external dedicated security validation team:

In some cases, it is good to have a 3rd party engagement for penetration testing of a product. This option completely outsources the service of penetration testing to an external company that specializes in this service.

Pros – The external security testers specialize in this type of work and may be dedicated to security testing. Outside source can provide the most objectivity. If needed, external security testing can provide certification to a specific level of security assurance. One beneficial way to achieve the most from this type of testing is to create a collaborative security test environment and work side by side to ensure the knowledge is shared amongst the development team and the security testing team.

Cons – This is most likely the costliest solution. Additionally, Intellectual Property (IP) protection is an issue and legal oversight with Non Disclosure Agreement’s NDA’s are needed to lower the risk to IP disclosure. This can also limit the amount of knowledge that can be provided to the 3rd party forcing a more black box or grey box approach to security testing …but a black/grey box approach may be the intent (pro) in some cases. Location and access to the testing environment for support may be limited for the customer, so validation of the testing environment may be difficult causing less assurance that tests are run with the most stable product(s) or version(s). There may also be less communication and collaboration between the testing team and the development team if the product is sent out for testing without expertise to assist the security testers in fully understanding the product.

To summarize, security testing through penetration testing is a valuable approach to validating security during development and before shipment of a product. If the risk of a product is valued high enough, it may be necessary for all of the proposed options to be utilized in some way so the suggestions provided here can be combined to ensure the best security solutions are planned in from the start. Even though some security testing can be performed at different stages in the development of a product, a pentest is most commonly planned for and scheduled before shipment of a product. But it is important to emphasize that security can never be appropriately and cost effectively “tested in” to a product after development.