Today, DevOps is enabling organisations to deploy changes to production environments at blazing speeds. A typical DevOps process flow through the following stages.
A developer writes code using any development environment of their choice and pushes it to a central source code repository.
The code is merged into a central repository management tool for the purpose of versioning.
The CI/CD server then pulls the code from the source code repository and packages the build artifacts/binaries.
These artifacts/binaries are then pushed onto a binary repository manager against the commit ID.
These artifacts/binaries are then pulled out to be deployed back into staging and production servers which are spun up using docker.
The service uptime is monitored regularly using a monitoring service.
For illustration purposes, we are going to use the environment and tools given in the image below to represent a typical DevOps process. The tools will vary for each organisation but the process depicted here and in the rest of the blog will be more or less the same.
This same process as seen on a “Jenkins Build Pipeline” would look like
What about Security?
Once the application is running in production or staging environment, we get a penetration test scheduled. Imagine a scenario where a production application is tested and a high-risk vulnerability, like a SQL injection, is discovered. In such a case, the team will be required to run the entire DevOps pipeline in order to fix a single SQL injection vulnerability. This is clearly not a viable option.
What if we bring security a little closer to the development cycle by embedding security testing within the DevOps pipeline? For e.g.: the SQL injection, in this case, could have been easily discovered at an early stage, with the help of a source code scanner and fixed even before packaging the artifacts.
As a consequence, we are shifting security more towards the left in the DevOps pipeline. And when we integrate more security controls/tools/processes in an automated fashion, our DevOps pipeline metamorphoses into a DevSecOps pipeline.
DevSecOps is the methodology of integrating security tools within the DevOps process in an automated fashion. DevSecOps is not just about tooling but also having the required knowledge about using those tools. This leads to a cultural change in the normal functioning of DevOps and the staff needs to be trained and asked to upskill themselves on the same. This allows them to collaborate more efficiently, thereby creating a “security culture”. This multicultural, multi-disciplinary automated security environment makes security everyone’s concern and not just of a single team which is also one of the major drivers of DevSecOps.
In a DevOps model, the places where security can be injected are shown in the image below. Once again, the tools chosen at each stage will vary for each organisation. In the examples below, we have used open-source tools and other free alternatives to depict that DevSecOps does not need to cost a fortune and even with the FREE alternatives and adequate training, amazing results can be achieved to mature the software security standard.
Pre-commit Hooks
Sensitive information such as the AWS keys, access tokens, SSH keys etc. are often erroneously leaked via the public source code repositories due to accidental git commits. This can be avoided by using pre-commit hooks like “Talisman” which checks for sensitive information in the files before commits or push activity.
Talisman – https://github.com/thoughtworks/talisman
Crass – https://github.com/floyd-fuh/crass
Git Hooks – https://githooks.com/
Git Secrets – https://git-secret.io/
Pre Commit – https://pre-commit.com/
Detect Secrets – https://github.com/Yelp/detect-secrets
Git Hound – https://github.com/ezekg/git-hound
Truffle Hog – https://github.com/dxa4481/truffleHog
Secrets Management
With automation, it is a common practice followed by developers and administrators to store credentials in configuration files, environment variables etc. for accessing various services. Storing credentials in the files or configuration can lead to exposure of credentials to an unintended audience. This can be segregated by leveraging secret management services like “Hashicorp vault”. This allows segregation of credentials on a separate level and every environment can pull credentials from a specific environment and use it programmatically.
Hashicorp Vault – https://www.vaultproject.io/
Torus – https://www.torus.sh/
Keywhiz – https://square.github.io/keywhiz/
EnvKey – https://www.envkey.com/
Confidant – https://github.com/lyft/confidant
AWS Secrets Manager – https://aws.amazon.com/secrets-manager/
Software Composition Analysis
In September 2017, Equifax, a popular credit rating agency was breached leading to leakage of 143 million credit card numbers, personally identifiable information (PII) and much more. The vulnerability that was exploited was found in the “Struts2 Web Framework” (missing security patch) on which the primary web application was built. Just like Equifax, a lot of organisations make use of open-source libraries/frameworks/solutions like WordPress, Magento, Drupal or even jQuery which are having new vulnerabilities being discovered every day.
Image Reference: https://snyk.io/blog/88-increase-in-application-library-vulnerabilities-...
For these reasons, it is necessary to perform an analysis of all the dependencies being utilised in the application and check them for vulnerabilities arising from missing security patches. For Java and .NET applications, we can make use of a tool called “Dependency-Check” which can be run before creating the builds to identify if any vulnerable software is being used in the application. A decision can be made depending upon the number and severity of vulnerabilities to either continue running the pipeline or fail it to fix the vulnerabilities. Below are a few tools to perform a software composition analysis for security vulnerabilities:
OWASP Dependency Check – https://www.owasp.org/index.php/OWASP_Dependency_Check
SonaType (Free for Open Source) – https://ossindex.sonatype.org/
Snyk (Free for Open Source) – https://snyk.io/
Bunder Audit – https://github.com/rubysec/bundler-audit
Rubysec – https://rubysec.com/
Retire JS – https://github.com/RetireJS/retire.js
Static Analysis Security Testing
Using automated tools to perform a security code review flushes out many low-hanging fruits like SQL injection, Cross-site scripting, Deserialization vulnerabilities and many more. For Java based applications we can make use of a tool called “FindSecBugs” which performs an in-depth analysis of the code(without giving out too many false-positives) and gives a comprehensive report for all the vulnerabilities that have been identified in the code. A decision can be made depending upon the number and severity of vulnerabilities to either continue running the pipeline or fail it to fix the vulnerabilities before the pipeline can continue. Below is a list of some open source tools that can be used for SAST purpose.
Image Reference: https://www.owasp.org/index.php/Source_Code_Analysis_Tools
Dynamic Analysis Security Testing
Web application scanners are an important part of doing a vulnerability assessment of a web application. A good majority of these scanners have API end-points or CLI access which can be leveraged to initiate scanning on target applications. OWASP ZAP is one such tool which can be used for initiating a web application security scan on an application in the QA/Staging environment to iron out a vast number of security misconfigurations such as disclosure of sensitive information in a backup file, insecure HTTP header etc.
OWASP ZAP – https://www.owasp.org/index.php/OWASP_Zed_Attack_Proxy_Project
Arachni Scanner – http://www.arachni-scanner.com/
Nikto – https://cirt.net/Nikto2
Security in Infrastructure as Code
Containerization solutions like Docker are very popular while building the infrastructure using simple few lines of code. “Docker Hub” is a popular public docker repository from where docker images are pulled to run containers. There have been various instances in which these docker images are laced with malware or are riddled with various vulnerabilities as shown in the image below. Hence, it is extremely important to scan these docker images that are being pulled from the public repository.
Image Reference:https://snyk.io/blog/top-ten-most-popular-docker-images-each-contain-at-...
One solution which provides a good insight into the security stature of the Docker containers/images is “Clair”. Clair scans the raw docker images and gives an exhaustive report highlighting the vulnerabilities that exist in the image. It is thus important to run a tool like Clair on the production docker images before deploying them in the infrastructure. Other tools to consider for this stage of DevSecOps are:
Clair – https://github.com/coreos/clair
Anchore Engine – https://github.com/anchore/anchore-engine
Dagda – https://github.com/eliasgranderubio/dagda
Open-Scap – https://www.open-scap.org/getting-started/
Docksan – https://github.com/kost/dockscan
Vulnerability Assessment (VA)
It is a general practice to perform vulnerability assessment on the production systems to identify various services running in the environment and the associated vulnerabilities.
While pointing a VA tool on the servers that have been created using Docker, it would execute the scan only on the service that is being exposed on that host. However, if we attach the tool to the docker network and then execute the scan, then it would give us a good picture of services which are actually running.
This can be done using various solutions like OpenVAS which can easily integrate into the pipeline.
OpenVAS – http://openvas.org/
DockScan – https://github.com/kost/dockscan
Compliance As Code
Organisations need to apply compliance controls to their IT infrastructure to abide by industry best practices and various regulations like PCI DSS, HIPAA, SOX etc. With “Infrastructure as Code” in DevOps, the production environment is never retained, it is always torn down and re-created again and hence it is strong requirement to test the updated/newly created environment after it has been setup. “Inspec” is one such tool which can help us in performing these tests as we only need to supply a ruby file containing the tests to be conducted in a very simple and lucid manner which is easy for every audit professional to write and code.
Inspec – https://www.inspec.io/
Serverspec – https://serverspec.org/
DevSec Hardening Framework – https://dev-sec.io/
Kitchen CI – https://kitchen.ci/
Vulnerability Management
The tools that we shall use to create a DevSecOps pipeline will generate plenty of vulnerabilities and each tool will have its own separate format. It becomes difficult to manage this data let alone track and remediate the vulnerabilities. Hence, vulnerability management solutions are at the core of a DevSecOps process where all tools are required to spool their data into those solutions so that it can be centrally managed, triaged, tracked and remediated.
“ArcherySec” is one such tool which not only has good integration with most of the tools mentioned above but we can also initiate scans such as Zap and OpenVAS through ArcherySec.
Below is a screenshot of ArcherySec dashboard showing the vulnerabilities aggregated across all the tools.
The following is a list of the tools that ArcherySec supports for pulling data
Other tools available for vulnerability management are:
ArcherySec – https://github.com/archerysec/archerysec
DefectDojo – https://www.defectdojo.org/
JackHammer – https://github.com/olacabs/jackhammer
Alerting and Monitoring
Production applications are always faced with new threats from unknown and unforeseen vectors. This can be mitigated by having an active intrusion monitoring and prevention solution. One such opensource solution is the “ModSecurity WAF(Web Application Firewall) which detects OWASP Top 10 vulnerabilities like SQL injection, Cross-site scripting etc. being attempted against the application. Below is a screenshot of ModSecurity in action.
ModSecurity WAF – https://modsecurity.org/
Conclusion
So far we saw the technical know-how of how DevSecOps would operate in an environment but just having the tools and techniques is not enough. DevSecOps requires a cultural change which promotes the “secure by default” culture. This could be achieved by creating security champions within each domain, increasing collaboration with the security team and so on.
Finally, DevSecOps is important for your DevOps model because it is the only way to handle “security at scale”.
Next Course of Action:
Claranet offers a full-day workshop on DevSecOps and AppSec for Developers to enable attendees to gain a comprehensive understanding of DevSecOps practice. If you would like to participate in the workshop or would like us to help you in implementing security within your existing DevOps pipeline by offering a bespoke solution for your needs, please contact us.