the concept of CI/CD system design in different stage from a startup to a scale-up team
In this article, we can learn the concept of CI/CD system design in different stage, and it is totally different in a startup team and in a scale-up team to design or upgrade your CI/CD pipeline and infrastructure system.
∘ Introduction
∘ Architecture Overview
∘ Design in startup v.s Upgrade in scale-up
∘ CI/CD Design and Implement in Phase 1 (Design in startup)
∘ CI/CD Design and Implement in Phase 2 (Design in startup to Upgrade in scale-up)
∘ CI/CD Design and Implement in Phase 3 (Upgrade in scale-up)
∘ Summary
Introduction
Remember, your end-user use this CI/CD system and it is great design. If your end-user don’t use this CI/CD system and it is a mess.
We recommend to see this book about Devops bible — The Phoenix Project — English, 鳳凰專案 — 繁中. The most important part in CI/CD system is communication, communication, and communication. In this article, We introduce our team’s design architecture on CI/CD system to maintain AWS cloud services and site services that are deployed to 11 cross-region sites.
Before designing our own CI/CD system , we communicate developer team and build up a simple project to demonstrate overall process and design concepts. This is very import step to align all tem members to the same page, and it would reduce time on misalign problem.
Moreover, we would go through the question on what is the evolution from a startup to a scale-up team in an organization (from 6 to 20+ peoples) and What is the key role for devops engineer in the growing team in following parts.
Architecture Overview
In this diagram, we design the whole process from devops process best practice (8 steps) and design the process from plan, trigger, continuous integration (CI), continuous delivery (CD), and continuous deploy (CD) — golden CI/CD process.
In different phase, the developer, stake holder or end-user have a vary of requirements to achieve their main goal, and devops engineer should fulfill these requirement without breaking the basic process. For example, the user want to know if this deployment is done, devops engineer should provide a email or slack notification after deploy step.
Devops process best practice (8 phases): you can refer The Eight Phases of a DevOps Pipeline (Jakob Pennington) for more detail about these 8 phase.
- Plan: The Plan stage covers everything about this product or system development.
- Code: The Code stage covers implementation on epic and user story level to achieve a user scenario.
- Build: The Build stage covers git commit trigger and build image or binary for all essential component of next testing stage. (Continuos Integration, CI)
- Test: The Test stage covers unit test, security check, and deployment to a stage environment. In this stage, we DO basic testing in dev/itg/stg environment. (Continuous Integration, CI)
- Release: The Release stage covers acceptable component or image for release purpose process. In company policy, team would ask manager to manually approve or pass it by code review a pull request. (Continuous Delivery, CD)
- Deploy: The Deploy stage covers a success build and deployment to target environment such QA or Production environment. Moreover, it release with different deployment policy such blue-green deployment, rolling deployment, or canary deployment.
- Operate: The Operate stage is now live and being used by the customers and cover the most import part feedback loop! The simple way to interactive with user is feedback function. Moreover, GA4 is a golden sample on e-commerce or blogger to acquire user’s category such as mobile or computer access and giving a feedback to this product.
- Monitor: The Monitor stage is to monitor the environment with key metric. Moreover, keep this system alive and be better. The monitor metric should build with two type — system metric and user metric. System metric make this service or product have minimum down time. User metric indicate the user experience and feedback improvement level.
Golden CI/CD process design (4 phases):
- Continuous Feedback (CF, 7,8,1 stage)- Continuous Feedback give more user requirement and generate suitable user story before writing any code. It was a exploration between product and requirement.
- Continuous Integration (CI, 2–3 stage) — Continuous Integration build source code to a images with qualified testing and scanning in pipeline. It ensure basic integration complete before delivery.
- Continuous Delivery (CD, 4–5 stage) — Continuous Delivery provide a stable images or product to be pushed to end-user or product environment.
- Continuous Deploy (CD, 4–6 stage) — Continuous Deploy provide a stable process to upgrade the product that would need complex configuration or upgrade multiple sites by manually configuration.
Design in startup v.s Upgrade in scale-up
Design in startup: The main goal in this stage should be get user input ASAP. Team should finish at lest 3–5 POC to proof this product benefit the end-user productivity or give more income. Don’t focus on high performance design, focus on user benefit design. This product life period would be 6–12 month in this stage and got to “Upgrade in scale-up”, or it would go begging.
Upgrade in scale-up: The main goal in this stage should consider refractory the parts of system and use the term product 2.0 to define this stage. Welcome, it pass the marketing testing of our end-users. This system would get lots of complain and feedback, and we should do next action to scale up the system. In this stage, it is very important on architect design meeting, all team member have to join that discussion to know how to turn a bundle of user requirements to a statable architecture with well-defined scope. This product life period would be 12–18 month in this stage and got to “Maintains Product”, or it would go begging.
CI/CD Design and Implement in Phase 1 (Design in startup)
In phase 1, the main goal would build a simple and readable CI/CD system for rapid POC development. Team doesn’t care about unit test rate or code quality too much, we just make sure in a basic quality level to meet the organization policy or cyber security requirement to release our product.
In the beginning 6 month, we spent most time on building team culture and understanding in devops culture and CI/CD process. For specific methodology, devops engineer share developer the boundary between different stages or how to implement the requirement in which stage, because some developer don’t have enough knowledge to know overall scope.
For overall system, we built the basic CI/CD system and devops process from zero, and we share 8 part implementation as below items. All thing is trad-off. Do NOT too care which tool is better. Delivery POC to end-user and build up devops team culture are the main goal at startup stage.
- Plan: Jira is the best tool to plan whole product backlog and sprint backlog. We lead developer to know all agile development process by Jira system and share the experience on stand up meeting, planning meeting, and retrospective meeting.
- Code: Github is our source control and most of developers are familiar the UI and github workflow.
- Build: Jenkins is the good solution to build our code to docker images to a artifactory.
- Test: In this stage, we share all idea about build container, unit test in nodejs or python, linter tool and code style, and product docker images in AWS ECR. In the end of this period, we ask developer to add simple E2E testing. Don’t block the developer in startup stage, and focus on POC delivery to end-user.
- Release: Jenkins build the complete image with testing and push to AWS ECR for pulling.
- Deploy: In cloud service, we use ECS task for auto deployment. In site service, we use ansible to configure site configuration and pull docker images to update site services. If you only have cloud service, that would be easy to manager by AWS service.
- Operate: We use cloud watch to collect cloud service logger to cloudwatch metric for debugging. For work loading, we put this logger system integration in the final parts.
- Monitor: For monitor, We use cloudwatch for cloud service and prometheus in site service for alarm notification.
CI/CD Design and Implement in Phase 2 (Design in startup to Upgrade in scale-up)
In phase 2, the main goal would refactor code and reduce technical debit on whole product with CI/CD system assistance. Team members should understand the developing process, write the acceptable code, and delivery the POC to the end user well. In this stage, about 3 month, team members should pay 70% on end-user requirements and pay 30% on refactoring and collecting more user requirements for 2.0 architecture preparation. When team cannot afford the work loading and too much customer complain, it is time to go for next stage Upgrade in scale-up. In this stage, the key metric are more team members joining and more end-user joining. If the scope increase the two time than original scope, the stage is coming.
In the 3–5 month, we spent most time on adopting the users’ and organization requirement to azure devops on planning stage and reduce the cost on infrastructure. Actually, it don’t have benefit in technical part due all thing are the same, but it benefit the end-user process. It is essential to discuss with team members for eliminating the questions. For overall system, we adapted our CI/CD system to fit users’ and organization policy and refactor the technical debit on our 1.0 system. In this stage, we call it “CI/CD 1.5”.
we share 8 part implementation as below items, actually, most of item are similar as 1.0 CI/CD
- Plan: We adapted JIRA story to AzureDevOps, and we spend about 2 month from planning to migrate all things to AzureDevOps. That include task like moving all tickets to AzureDevOps, new dashboard to show performance and key metric for developing, and turn team’s workflow to another system. It is a big engineering.
- Code: Github is our source control and most of developers are familiar the UI and github workflow.
- Build: Jenkins is the good solution to build our code to docker images to a artifactory.
- Test: In this stage, we share all idea about build container, unit test in nodejs or python, linter tool and code style, and product docker images in AWS ECR. In the end of this period, we ask developer to add simple E2E testing. Don’t block the developer in startup stage, and focus on POC delivery to end-user.
- Release: Jenkins build the complete image with testing and push to AWS ECR for pulling.
- Deploy: In cloud service, we use ECS task for auto deployment. In site service, we use ansible to configure site configuration and pull docker images to update site services. If you only have cloud service, that would be easy to manager by AWS service.
- Operate: We use cloud watch to collect cloud service logger to cloudwatch metric for debugging. For work loading, we put this logger system integration in the final parts.
- Monitor: For monitor, We use cloudwatch for cloud service and prometheus in site service for alarm notification.
CI/CD Design and Implement in Phase 3 (Upgrade in scale-up)
Summary
We learn from the devops journey that “DevOps is a culture for a team and people, and CI/CD is to implement the concept to service user for more rapid experience and explore the system unstable part in quickly iteration.” We share these experience for your reference
