The BC Gov's Private Cloud Platform as a Service (PaaS) is a reliable and security-compliant^1 application hosting platform for running government services in the on-premises private cloud. It is available for use by ministries, agencies and crown corporations working with the Government of British Columbia. The Private Cloud PaaS is powered by RedHat's OpenShift Container Platform (OCP) technology, and is hosted in the government's Data Centers; Kamloops (main operations) and Calgary (disaster recovery only).
BCGov’s Private Cloud PaaS offers tools to help product teams in the BC Government that are building online services for citizens, develop and run modern, cloud native software applications. We, at the Platform Services Team in BCDevExchange, Ministry of Citizen Services, maintain and secure the infrastructure, so your team can focus on building and improving your applications!
Product teams are offered a choice of two hosting tiers on the Private Cloud Platform - Silver and Gold. The Silver tier provides application hosting on the "Silver Kamloops" production cluster. The Gold tier provides application hosting across the pair of “Gold Kamloops” and “Gold Calgary” production clusters.
|Who should use it?||Recommended for the majority of government services supported by Product teams with junior to intermediate DevOps skills||Recommended for business mission critical government services supported by a fully funded Product team with advanced DevOps skills|
|How much does it cost?||Free in 2021/2022 fiscal. Cost recovery model may be implemented in 2022/2023 fiscal as part of the Enterprise Services cost review directed by Treasury Board, and will continue to consult with clients in that space on whether there will be cost recovery for Private Cloud, as well as what it will look like if there is one.||Free in 2021/2022 fiscal. Cost recovery model may be implemented in 2022/2023 fiscal as part of the Enterprise Services cost review directed by Treasury Board, and will continue to consult with clients in that space on whether there will be cost recovery for for Private Cloud, as well as what it will look like if there is one.|
|Maintenance Schedule||Private Cloud PaaS upgrades and patches will be applied in this production cluster quarterly after testing in LAB clusters||All Private Cloud PaaS upgrades and patches are applied in Silver first; these will be applied quarterly in Gold a few weeks later.|
|High Availability||App horizontal scaling within Silver cluster, set up and managed by the Product team.||App horizontal scaling within Gold Kamloops cluster AND a requirement to set up a geographic failover to Gold Calgary, both set up and managed by the Product team.|
|Network Services||Standard OpenShift routing. Custom TLS certificates are mandatory for Product Teams to bring with the application.||Standard OpenShift routing, non-HTTP ports exposed to the DMZ for replication, and Global Site Load Balancing which is required for failover. Custom TLS certificates are mandatory for Product Teams to bring with the application.|
|Platform Service Availability Level||90% for single-node application deployments^2 99.5% for multi-node application deployments^3||99.95%^4 for multi-node application deployments with geographic failover^5|
BC Gov's Private Cloud as a Platform Service includes:
- a set of four project namespaces with self-serve developer access: tools (for development of lifecycle support tools such as CI/CD pipelines, automated testing, and code quality tools), dev, test, and prod - each corresponding to a deployment stage in the application life cycle,
- "small" project resource quota by default (a bundle of CPU, RAM and storage resources) with ability to upgrade to "medium" and "large" as required. (See the project resource quota sizes here ),
- OCIO Standard Backup and restore services for application data. See more detials here,
access to the DevSecOps tools to help teams build "Secure by Design" applications:
- Sysdig App Monitoring Service allows building robust dashboards for applications to monitor their health, availability and resource usage
- Artifactory Repository Service provides access to a trusted and secure repository for storing images, packages, libraries and other artifacts
- Vault Secret Management Service provides a secure way to store and manage credentials, API tokens and other sensitive app information
- AQUA Container Scanning Service allows teams to scan their running containers to find security vulnerabilities
- EnterpriseDB HA service for PostgreSQL – a vendor-supported product for running highly available PostgreSQL clusters (a product team must purchase their own license in order to use this service)
- CrunchyDB HA service for PostgreSQL (coming in Dec 2021 for early access in Silver) – an open-source version of the Postgres operator from Crunchy Data for running highly available PostgreSQL clusters
Other security features include:
- The majority of Private Cloud PaaS maintenance has no/minimal impact on the applications configured as multi-node deployments running on it and is run during business hours as required in a containerized environment.
- Multi-tenant hosting model in OpenShift and the build-in software defined network using Kubernetes Network Policies provides isolation between teams' environments so that they can't read or change each other's code, data or logs
Developers receive self-serve access to create and manage application's network security rules for their own apps. The Platform Community has a vast collection of design patterns that follow best security practices, for building integrations between the OpenShift applications and external systems. In addition, new technology and design patterns are being developed in partnership with ministries to ease this work even more.
- Authenticated SSH access to application containers to debug problems i.e. rsh into a pod
- Platform and application namespace access through org-restricted GitHub IDs (2FA required)
- Single sign-on service for end-user authentication for apps through the BC Gov's Single Sign-On Service based on KeyCloak (https://oidc.gov.bc.ca)
- The Private Cloud PaaS is piping core Platform log files into the central Security Information and Event Management (SIEM) system in OCIO for additional forensics and security audits. Application log shipping and developer access to SIEM are coming soon.
- Information Security Classification: Protected B. The Private Cloud PaaS is approved for storage of Protected B information at rest. However, if an application is designed with appropriate controls to protect data in transit, including the use of their own TLS certificate, an OpenShift application may pass Protected C information via APIs to a system component off-cluster (i.e. legacy system). However, in this circumstance, DevOps teams should connect with their Ministry Information Security Officer (MISO) prior to collecting/using/storing Protected C information. Once the VMWare NSX-T solution implementation and testing has completed, the information security classification level for the Private Cloud PaaS will be re-assessed.
The Platform Services Team with the help from Platform Operations and Data Center support teams monitors the infrastructure, OpenShift Container Platform and the critical DevOps Security services such as Vault and Artifactory, 24/7. All critical incidents that relate to the availabilty of the Openshift 4 Platform and/or its critical services outside of the regular business hours should be reported to Shared Services BC Service Desk at 250-387-7000 (aka 7-7000).
We, at the Platform Services Team, also manage the patching of the platform's operating systems and infrastructure components. When we update the platform, we use zero downtime maintenance, so this won't interfere with the running of your application if and only if your application is designed for resiliency.
All non-critical Platform services and DevOps Security services are supported during business hours 9am-5pm Mon-Fri excluding statutory holidays.
The Openshift 4 Platform Change Management process can be found in the following flow chart.
Access to the Private Cloud Platform is available for BC government projects that meet the following criteria:
- Have Executive Sponsorship Your executive understands what is required to support continuous service improvement and is committed to resourcing your project so that it can be continuously maintained and improved even after it's been built.
- Are Open Source Your project is based on open source code, with custom code hosted in one of the BCGov GitHub organization repositories.
- Have a Dedicated Agile Teams Your project will be supported by a fully funded team that follows an agile methodology, with explicit roles such as DevOps specialist, Scrum Master and Product Owner — ideally with one or more of these roles filled by dedicated staff. Note: we require that the role of the Product Owner be filled in by a permanent government employee.
- Collaboration Your team is committed to participate in and contribute to the B.C. government's open development community.
Access to the Private Cloud Services is managed through a central application, the Openshift 4 Platform’s Product Registry. This application that we manage as a part of the Platform Service uses automation for the creation of your hosting space. A team can have an hosting environment created following an approval in as little as 10 minutes. If you are net new to this service, you will need to have an onboarding session with us first. Please reach out to Olena Mitovska, Product Director of the Private Cloud PaaS to get that started. If you have any questions around the Platform Project Registry, she can answer those too. If your team meets the criteria above and is looking to leverage the Private Cloud PaaS, we are looking forward to welcoming you.
Much like we know you pride yourselves on the services you provide in your application development as part of your Product Team, we pride ourselves on the Private Cloud PaaS we provide. Real-time uptime information for the DevOps Security services and the OpenShift clusters that we support can be found on the PaaS Reliability Dashboard at https://status.developer.gov.bc.ca , to compliment the SLA objectives above. More information about the Reliability Dashboard and what powers it underneath the hood, can be found here
As a member of the Private Cloud, you must join the BCDevExchange, the Province of British Columbia’s vibrant developer community, also known as the Platform Community. We leverage that community as part of our support model which is described below. The Platform Community relies on Rocket.Chat, an open-source communication platform, to connect and stay in touch with each other and us, the Platform Services Team. You can read about how to join the RocketChat here.
We also provide additional communication channels for you to subscribe to for staying up to date on the Private Cloud PaaS offering and learn about major upcoming changes ahead of time. Please sign up to the Platform Updates Subscription service to join our email distribution list. In addition, once you join the community and service offering, we also encourage you to join our monthly Private Cloud PaaS Community Meetup Series where we share platform improvements, demos from ourselves and the community, and opportunities to engage with our service design research team. Please, ping Olena Mitovska to get a meeting invite.
The Platform Services Team follows a team of teams model made up of the Platform Experience Team, and Platform Operations Team. Have a look at our org chart here. While the Platform Services Team manages infrastructure, OpenShift Container Platform and the Platform critical services as part of the Private Cloud PaaS, the Product Team bears the responsibility for the functionality and operations of their application(s) hosted on the Platform.
|Resource||Responsibility for: Operational Support, Monitoring, Troubleshooting and Access Management|
|Application data||Product Team|
|Application support and operations||Product Team|
|Application network security||Product Team together with their MISO|
|Application monitoring||Product Team|
|Efficient use of Platform Resources (CPU, RAM and Storage) by the Application||Product Team|
|Application Integration with DevSecOps tools||Product Team|
|Application Integration with KeyCloak SSO Service||Product Team|
|Platform Physical Infrastructure and OpenShift Software||Platform Services Team: Operations Team (DXC/Advanced Solutions)|
|DevSecOps Tools (AQUA, Artifactory, Sysdig, EnterpriseDB Operator and Vault)||Platform Services Team: Platform Experience and Platform Operations Teams|
|BC Gov SSO Service||KeyCloak SSO Support Team (contact Zorin Samji (SSO Product Owner) for more details)|
Note:We are experiencing a critical situation on the Private Cloud Platform at this moment with a high amount of CPU, RAM and Storage resources being reserved but not used and if not mitigated, this situation will force us to freeze onboarding of new apps onto the Platform in the near future until we find a way to motivate product teams to use resources more efficiently as described in the Platform App Resource Tuning Guidelines. If you an existing product team, please review the resource allocation for your app and make sure they are aligned with the Guidelines.
If you have any questions about the shared responsibilities above, please contact Justin Hewitt, Sr Director of DevOps Platform Services at Justin.Hewitt@gov.bc.ca.
The diagram below shows the responsibilities that different groups within the government have in supporting the OpenShift 4 Platform, its infrastructure and the Ministry apps .
The Platform Community includes 2,000+ participants across all ministries of BC Government and is a mix of government employees and contracted resources. While the Platform Services Team doesn’t provide direct application development support, the strong community of Platform users helps each other solve app-level questions. Read more about the Community-based User Support Model here
The Platform team uses the open-source communication platform Rocket.Chat to connect with each other and help each other troubleshoot issues. Refer to the chat channel convention to find an appropriate channel to post your question.
If a team believes that an issue with the Private Cloud PaaS and not with the app itself, they can engage with the Platform Services Team in #devops-sos channel (if it is an urgent and critical production issue) or in #devops-operations channel (for non-urgent and non critical production issues) in RocketChat. Teams can also reach out to the Platform Community with general questions in #devops-how-to channel.
We currently do not offer a Tiered Service Desk model on the Private Cloud PaaS, RocketChat is the primary communication tool to contact Platform Services Team to ask questions and get help.
Periodic internally-delivered training is provided by the DevOps Platform Services team. The internal training schedule is available here. Commercial OpenShift training is also available from Red Hat. For details on the commercial training, contact Olena Mitovska, Product Owner for Platform Services.
Welcome to the DevOps Platform and to the Platform Community!
Love, Platform Services Team, xo