DevHub

beta

Topic

Platform Services Security

Modern day resources for securing your applications and services.

BC Government OpenShift DevOps Security Considerations

DevOps Security Tools and Capabilities

Summary

There are a number of tools available to developers working on the OpenShift platform to help ensure the confidentiality, integrity and availability of data within those systems. This is an overview of those tools, with links to specifics on each of the resources.

Tools & Capabilities

OpenShift Platform Security

Service definition - Silver/Gold - https://developer.gov.bc.ca/Getting-Started-on-the-DevOps-Platform/BC-Gov-PaaS-OpenShift-Platform-Service-Definition

Our Silver Service is our standard DevOps platform offering, with on-cluster resiliencey based on application design.

Our Gold Service is our enhanced DevOps platform offering, with replication to a secondary cluster for disaster fail-over purposes.

Please take note of the Shared Responsibility Model. While the Platform Services Team manages infrastructure, OpenShift Container Platform and the Platform critical services as part of the Private Cloud PaaS, the Product Team bears the responsibility for the functionality and operations of their application(s) hosted on the Platform.

Specific details on OpenShift specific secuirty controls can be found here:

https://www.redhat.com/rhdc/managed-files/cl-openshift-security-guide-ebook-us287757-202103.pdf

  • these are highlighted as part of the OpenShift STRA.

Penetration Tests The platform services team outsources for a penetration test annually to ensure the services we provide are configured to protect confidentiality, integrity and availability. Peneteration tests are procured through the pre-qualified list of vendors (https://www2.gov.bc.ca/gov/content/governments/services-for-government/bc-bid-resources/goods-and-services-catalogue/im-it-security-services).


Privacy

A Privacy Impact Assessment has been completed for the OpenShift Container Platform service.

Personal Information upto and including Protected B Information Security Classification may be stored on OpenShift.


Critical Systems Standard

We are very close to obtaining critical systems standard compliance.
Documentation is in final stages of review for submission.


Platform Tools Security Assessments

Many of the platform tools have completed security assessments. These include:

  • OpenShift v4.x and GitHub (Public)
  • KeyCloak
  • Aqua
  • Artifactory
  • Sysdig Monitor
  • Just Ask!
  • Rocket Chat
  • Vault

The following security assessments are underway:

  • Mautic
  • OCP Application Resource Tuning Advisor (in sign-off)
  • GitHub Enterprise (Cloud security schedule review underway)
  • 1Password (SoAR complete, Cloud security schedule review underway)
  • Platform Registry

The following security assessments are planned:

  • Stack Overflow
  • Cert Manager for OpenShift
  • GitGuardian
  • KeyCloak Realm Registry
  • OCP App Assessment Tool
  • Platform Security Dashboard

For specifics, please contact the platform security architect, nick.corcoran@gov.bc.ca.


Platform Registry

Here, we maintain a listing of all projects with deployments on each OpenShift cluster. platformregistryexample

https://registry.developer.gov.bc.ca/public-landing?redirect=/dashboard

While access to the registry is currently limited to the OpenShift Platform Services team (full view) and Product Owners/Technical Leads (limited view), we are working on creating roles for Ministry security staff to consume as well. Until then, you can contact nick.corcoran@gov.bc.ca for details.


Communications

Community sharing, alerts and discussions take place on Rocket Chat, which we host as an app on OpenShift. Authentication via IDIR or GitHub (in BCGov org or invited by an existing member).

Mautic

Mautic has been implemented to allow for subscription based communications for the DevOps community.


Access Management

Access to OpenShift is brokered through our OpenShift SSO Service (currently leveraging KeyCloak).

GitHub has been the primary authentication to date on OpenShift, however we are in the process of introducing IDIR (via Azure AD). Both of these options require an account with 2FA/MFA enabled.
GitHub - All clusters
IDIR - KLAB, Silver (Gold, GoldDR to come in early 2022)

Cluster roles are managed either in private GitHub repositories (in the bcgov-c org) or through direct role bindings within a namespace. We are investigating third party tools to help improve the user management experience.

Platform Services Roles and Responsibilities can be found here:

The Platform Services team maintains an Access Control Policy for all platform tools.


Kubernetes Network Policies

Network policies help the platform and project teams to better control communications between components. While KNPs only apply as INGRESS rules (not egress), they help to improve our overall security posture. KNPs only apply to on-cluster communications (i.e. between pods in a namespace, or between namespaces). For off-cluster communications, hosting is investigating a VMWare tool called NSX-T.

This resource is a little out of date, as it was orginially created for migration of applications from our OpenShift 3.11 cluster to our OpenShift 4.x cluster, but the details on KNPs is still relavent.

More details on KNPs can be found here: https://kubernetes.io/docs/concepts/services-networking/network-policies/


Pipeline Templates (includes static and dynamic analysis)

In order to reduce effort in implementing secure tools into a build pipeline, we have developed pipeline templates that include components for build, aas well as static and dynamic vulnerability scanning.

Here is a representation of what an application build pipeline should look like: PlatformSec drawio

The pipeline templates above make it easier to include the tools described below:


Container image scanning (Aqua, Xray)

Image scanning comes in 2 forms - 1 active (Aqua), 1 passive (XRay).

Aqua:

This tool allows us to scan image registries and running containers for image vulnerabilities. It also allows us to create policies at build, deploy, and runtime.

We are also workign with teams on integrating Aqua container image scanning into pipelines. Alternatively, Aqua image scans may be requrested by contacting Nick Corcoran via email (nick.corcoran@gov.bc.ca) or on Rocket Chat. Nick can walk through results with the team if desired.

XRay:

An addon capability to Artifactory, XRay scans images and other artifacts for component vulnerabilities. Anyone with access to an image or artifact within Artifactory can see the XRay tab as part of the image/artifact details, and see what vulnerable components lie within, and what version will correct that deficiency.


Container runtime security

We currently have runtime policies in place for the following using Aqua: aqua_enforce

Additionally, OpenShift uses CoreOS and the CRI-O container engine.


TLS Certificates

OpenShift uses a wildcard certificate for the majority of cluster communications security. This should be sufficient for dev and test workloads, but for production workloads, each team is required to obtain a dedicated TLS certificate from the Access & Directory Management Services (ADMS) team.
Note: by default, the wildcard will be used to protect project workloads. The Platform Services team worked through the wildcard issuance requriements for use on the OpenShift clusters. Obtaining a dedicated TLS cert is currently a manual process.
Details on these processes can be found here: https://ssbc-client.gov.bc.ca/services/SSLCert/documents.htm

Pre-requisites: Generate a .csr for each site:

Ordering Process:

  • Business area creates/submits order
  • Ministry Service Desk reviews order, creates iStore Number, sends to EA for Approval
  • EA Approves
  • Order is sent to DXC for fulfillment
  • Once order is fulfilled/shipped by DXC, Ministry Service Desk sends 'Completed Order' notification to business area

TLS_Order


Secrets Management

OpenShift Secrets:

This 'secrets' store should actually only be used for configurations. Values are encoded as base64 and NOT encrypted. However, these 'secrets' can only be accessed with a role to a given namespace with permission to access them. Additionally, the physical etcd volume, where OpenShift secrets are stored, is encrypted.

Vault:

The preferred secrets management tool, Vault was recently launched for team use on OpenShift.


GitOps/Cluster Configuration Management

Argo CD provides a GitOps capability for sync'ing a Git repository with an OpenShift configuration (platform or application).


Application Programmable Interface (API) Management

The Data BC team hosts an API Gateway for use by other government clients. Details can be found here:


Application Resource Tuning Advisor and App Assessment Tool

Resource Tuning Advisor

App Assessment Tool


Logging/Monitoring (EKS, Kibana, Graphana, Sysdig Monitor, SIEM, Uptime, Status)

The Platform Services team provides a number of tools to help ensure our platform and applications are behaving as expected, while allowing us to investigate anomolies.

OpenShift UI: Within the OpenShift interface, project teams can view logs associated with a given pod through the Logs tab.
logging_ui

Kibana:

This tool provides a more wholistic view of logs for an application or at the platform level, as well as providing visualization and alerting capability.

Sysdig Monitor:

This tool allows our platform admins and platfrom teams to build monitoring dashboards.

Security Information and Event Monitoring (SIEM):

All cluster logs (system, audit, container) are regularly sent to the Security Operations SIEM environment.
Retention is set as follows:

  • System, Container logs - 2 months
  • Audit logs - 13 months

We are currently working on Azure AD integration (via KeyCloak) and user role mapping based on OpenShift namespace access.

Uptime Robot (currently being replaced with Uptime.com for improved SLAs).

This tools help us to observe platform service availability:


Backups

OpenShift:

GitHub:


Change Management

Planning for platform and service changes is documented on the Platform Services ZenHub board.

Strategic level changes are communicated to the DevOps community at regular Community Meetups, as well as to executive groups across government.


GitHub

GitHub is the primary git repository for platform application code. There are some exceptions that use privately hosted GitLab or other source code repositories.

Here is a summary of the GitHub organizations we own and their purposes:

  • bcgov - main developer git repository for platform application code and/or public sharing.
  • bcgov-c - main private git repository used for cluster configuration management and non-public projects.
  • bcdevops - alternate git repository for platform application code. Membership required for access to OpenShift.
  • bcgov-platform-services - git repository for platform services team

These resources are available on the developer hub:

GitHub Apps

Teams may request GitHub apps to be associated with their own or all projects in a GitHub organization. These requests are reviewed by a platform administrator for validity and scope. These are also shared with Ministry security staff to ensure they are acceptable for connection.

GitHub Enterprise

We are currently piloting the use of GitHub Enterprise.


Other considerations

Payment Card Industry Compliance (PCI-DSS)

Our OpenShift implementation is NOT PCI-DSS compliant. If you wish to host an application on OpenShift that needs to perform financial transactions, please refer to the following: https://developer.gov.bc.ca/Payment-Card-Processing-for-OpenShift-Applications Some teams have decided to host PCI-scoped applications on-prem (non-OpenShift) or on a cloud based service (AWS, Azure, etc) to avoid linkages with government systems not under their control.

Training/Support

The platform services team provides training to onboarding teams, as well as support for issues experienced. Ministry staff that work with devops teams are also encouraged to attend training.

Training:

Support:

App security self assessment:

Best practices for building apps on OpenShift:

Contact For all other matters concerning security on the OpenShift Container Platform, please contact nick.corcoran@gov.bc.ca.

  • Create an Issue

Platform Services Security
Content

  • home
  • disclaimer
  • privacy
  • accessibility
  • copyright
  • contact us
  • Government Of BC