Introduction
Continuous Integration/Continuous Delivery (CI/CD) is a cornerstone of modern app development. It helps foster collaboration, accelerates development speed, reduces error rates, and boosts software quality. Two leading tools, CircleCI and Jenkins, dominate the CI/CD landscape, each with its unique features and capabilities. This comprehensive guide will deep dive into a detailed comparison between the two and explore their integration with pCloudy, a cloud-based app testing platform.
I. CircleCI: An Overview
1.1 What is CircleCI?
CircleCI is a powerful, cloud-based CI/CD tool. Renowned for its ease of setup and use, CircleCI stands out with its outstanding Docker support, automatic parallelization, and optimization of tests for speed.
CircleCI primarily operates based on a YAML file (.circleci/config.yml) located in your project’s repository, which defines your CI/CD pipeline. With a robust API, debugging tools, and solid caching strategies, CircleCI can significantly optimize your build processes.
1.2 Key Features of CircleCI
- Cloud-Based: No software to install or servers to maintain.
- Performance Optimization: Auto parallelization and test splitting for faster build times.
- Docker Support: Extensive support for Docker and Docker Compose.
- Debugging Tools: SSH access to failed builds for debugging.
- Flexible Pricing: Free tier available, with flexible plans for scaling.
1.3 Pros
- Ease of Setup: CircleCI is cloud-based and easy to set up without the need for server maintenance. It’s simple to configure using YAML files and to get it up and running is generally more straightforward than Jenkins.
- Parallel Execution: CircleCI provides a strong platform for executing jobs in parallel, thereby reducing the total time taken for the CI/CD pipeline.
- First-class Docker Support: CircleCI provides superior Docker support, which is essential in modern development environments.
1.4 Cons:
- Limited Customization: Compared to Jenkins, CircleCI provides fewer customization options. This is a downside for teams needing to tweak the environment to fit specific needs.
- Cost: CircleCI can become costly for larger teams or complex projects, especially when compared to Jenkins, which is free and open source.
2.1 What is Jenkins?
Jenkins is an open-source automation server that’s immensely popular for implementing CI/CD pipelines. Its extensive plugin ecosystem, flexible and customizable nature, and the ability to run on various operating systems make it a favored choice among many DevOps teams.
Setting up Jenkins involves creating projects and configuring ‘Build Triggers’ and ‘Build Steps.’ The extent of this customization is what lends Jenkins its flexibility, but it could also make the learning curve steeper for beginners.
2.2 Key Features of Jenkins
- Extensive Plugins: Over 1,500 plugins to integrate with other software.
- Distributed Builds: Ability to set up multiple build servers for workload balance.
- Pipeline-as-Code: Pipelines are defined within the source code repository.
- Automation: Automated builds, testing, and deployment.
- Open Source: Free to use and backed by a strong community.
2.3 Pros
- Highly Customizable: Being open-source and having a rich plugin ecosystem, Jenkins allows for a high level of customization to fit your CI/CD pipeline’s needs.
- Large Community: Jenkins has a large and active community, providing abundant resources for learning and troubleshooting.
2.4 Cons
- Complex Setup: Jenkins can be complex and time-consuming to set up and configure. It requires server setup and maintenance.
- Dated Interface: Jenkins’ UI is not as modern or user-friendly as CircleCI’s.
III. CircleCI vs. Jenkins: A Head-to-Head Comparison
The central debate between CircleCI and Jenkins is not one that can be resolved through a single winner-takes-all declaration. Instead, let’s scrutinize the key points of distinction to understand how each fares in certain areas.
Installation and Setup
CircleCI: CircleCI is a cloud-based solution that eliminates the need for installation. Simply connect CircleCI to your version control system (GitHub/Bitbucket), and it automatically detects and builds your projects. This streamlined setup process reduces complexity and saves time.
Jenkins: Jenkins is a self-hosted solution that requires installation on a server. While it offers the flexibility to run on a variety of operating systems, setup can be more complex and time-consuming. However, Jenkins offers the advantage of being highly customizable and adaptable to specific infrastructure requirements.
Configuration and Maintenance
CircleCI: Projects in CircleCI are configured through a YAML file, known as “circle.yaml.” This file defines the tasks and workflows, making it easier to manage and maintain. CircleCI’s automatic parallel task configuration further simplifies the setup process.
Jenkins: Jenkins requires extensive configuration through its web-based interface. However, its “Pipeline as Code” feature allows you to define and manage workflows using code, reducing the manual configuration effort. This flexibility enables highly customized CI/CD pipelines tailored to specific project needs.
Scalability and Performance
CircleCI: As a cloud-based service, CircleCI scales seamlessly and offers excellent performance. Builds are automatically balanced and queued, ensuring efficient utilization of resources. This scalability is particularly advantageous for teams working on projects with fluctuating build demands.
Jenkins: Jenkins can scale, but it requires manual configuration of additional nodes to distribute the workload. Performance depends on the capacity and configuration of the server infrastructure. While Jenkins can handle large-scale projects with the right setup, it may require more maintenance and monitoring to ensure optimal performance.
Integration Ecosystem
CircleCI: While CircleCI has a growing ecosystem of integrations, it may not have the same breadth as Jenkins due to its relative newness. However, it provides seamless integration with popular version control systems like GitHub and Bitbucket, as well as a range of other developer tools and services.
Jenkins: Jenkins has been in the market for a longer time and boasts a vast ecosystem of plugins and integrations. It supports a wide range of tools and technologies, making it a versatile choice for integrating with various systems. The extensive plugin ecosystem allows for easy integration with popular tools, services, and infrastructure providers.
Community and Support
CircleCI: CircleCI has a supportive community and offers documentation, tutorials, and community-driven support. While it may not have the same level of community engagement as Jenkins, it provides valuable resources for troubleshooting and getting started with the platform.
Jenkins: Jenkins has a large and active community of users, contributors, and developers. This active community translates into extensive documentation, tutorials, and community-driven support. You can find answers to most questions and troubleshoot issues easily due to the wealth of resources available.
Cost and Pricing Model
CircleCI: CircleCI offers a range of pricing plans, including free tiers for small projects. Their pricing model is typically based on the number of concurrent builds and additional features required. This allows organizations to choose a plan that aligns with their needs and budget.
Jenkins: Jenkins, being an open-source tool, is free to use. However, it’s important to note that the costs associated with Jenkins may arise from server infrastructure, maintenance, and additional plugins or integrations. These costs should be considered when evaluating the overall expenses.
Flexibility and Customization
CircleCI: CircleCI provides a simpler configuration process and is designed for ease of use. While it may not offer the same level of flexibility and customization options as Jenkins, it provides a streamlined CI/CD experience that caters to many project requirements.
Jenkins: Jenkins is renowned for its flexibility and customization options. With its vast plugin ecosystem and the ability to define pipelines as code, Jenkins allows for highly tailored CI/CD workflows. This flexibility makes it a popular choice for organizations with complex requirements.
In addition to the above aspects, let’s summarize the other key differences between CircleCI and Jenkins:
| Aspect | Jenkins | CircleCI |
| Build Management | Builds managed via UI; configurations stored in files or Jenkins database. Less convenient for sharing configuration data. | Tasks defined in a single YAML document (circle.yaml), easily shareable like other source code repositories. |
| Server Requirements | Requires dedicated server and manual maintenance. | Operates on a cloud-based platform, automatically executing code in fresh containers. Requires less maintenance. |
| Debugging | Debugging process can be complex, often requiring manual DevOps testing. | Offers simple debugging with SSH and automated DevOps testing features. |
| User Interface | UI can be slower and less responsive due to local server dependency and reliance on numerous plugins. | Provides a continually improving UI with built-in support, enhancing speed and responsiveness. |
| Docker Workflow | No inherent support for Docker workflows; developers need to install separately. | Built-in Docker support via the services section of the circle.yaml script. |
| Parallel Builds | Supports parallel tasks through multi-threading. | Offers built-in support for parallelism, configurable through developer settings. |
| Data Security | Ensures security of confidential files and encrypted data through plugins and credentials. | Provides limited settings for encrypted files, with most data accessible to developers. |
After evaluating these categories, CircleCI and Jenkins demonstrate distinct strengths and considerations. CircleCI excels in its cloud-based deployment, simplified configuration, and scalability. On the other hand, Jenkins stands out for its extensive plugin ecosystem, customization capabilities, and the support of a vibrant community.
Ultimately, the choice between CircleCI and Jenkins depends on various factors such as project requirements, team expertise, scalability needs, and budget considerations. It’s recommended to thoroughly evaluate these factors and potentially conduct a trial or proof of concept to determine which tool aligns best with your organization’s needs.
Understanding pCloudy: Uniting CI/CD Tools with Cloud-Based Test Automation Platform
Having analyzed the contrasts between CircleCI and Jenkins, let’s delve into how to integrate a test automation tool with pCloudy for DevOps testing.
DevOps testing, or Continuous Testing, is a blend of CI/CD and test automation. The goal is to test every build, whether it’s committed, integrated, deployed to a new environment, or finally launched. This is where integrating CI/CD tools with pCloudy’s platform can accelerate your apps time-to-market. Let’s see how it’s done.
Integrating CircleCI with pCloudy
CircleCI enables you to carry out automated testing on your code before merging any changes. You can incorporate various testing tools and frameworks like Mocha, Jest, pytest, XCTest, JUnit, Selenium, and more. In the event of failed tests, CircleCI provides access to test output data for debugging purposes. By storing your test data within CircleCI, you can leverage Test Insights and parallelism capabilities to identify ways to optimize your pipelines further.
pCloudy has integrated CircleCI so that every commit is automatically passed through the automated pipeline before it is pushed to the relevant feature branch. With pCloudy, you can carry out automated testing of your Native and Web applications, ensuring that your development code functions seamlessly across a comprehensive online Appium grid comprising over 5000+ device-browser combinations that operate on the cloud.
Prerequisites
In order to conduct Mocha tests using pCloudy, you must have the following prerequisites in place:
- Global Dependencies
- A Git or GitHub repository
Getting Started
You must first download and install node.js and node package manager or npm. For installation of node.js using homebrew, execute the given command in your terminal:
$ brew install node
Plain text
Copy
In case you already have npm installed, it is advisable to update it to the most recent version. You can do this by running the following command in your terminal:
$ npm install npm@latest -g
Plain text
Copy
Overview
To incorporate pCloudy with CircleCI, you must include the configuration file within the .circleci/config.yml directory in your feature branch on GitHub, and specify all the required configurations to conduct the tests. With every commit, CircleCI will execute the build (#config.yml), while simultaneously running the test on pCloudy My Active Sessions.
It is essential to set up your repository with CircleCI to enable the running of build steps.
Below is the sample config file for Integration with Mocha-sample-test :
.circleci/config.yml this directory should be added in the root directory of the branch
And below is the content for config.yml to run the build for test
version: 2 # Specifies the version of CircleCI being used.
jobs:
build:
name: pCloudy # Specifies the name of the job as “pCloudy”.
docker:
– image: cimg/node:20.0.0 # Defines a Docker image (cimg/node:20.0.0) for the job to run within.
steps:
– checkout # Check out the source code.
– run: npm install # Installs dependencies using npm.
– run: npm run android # Executes the “npm run android” command.
Plain text
Copy
The given code shows the desired capabilities needed to execute the Sample-test.
capabilities = {
“browserName”: “”,
“pCloudy_Username”: “Enter-Email”,
“pCloudy_ApiKey”: “Enter API-Key”,
“pCloudy_ApplicationName”: “pCloudyAppiumDemo.apk”,
“pCloudy_DurationInMinutes”: “10”,
“pCloudy_DeviceFullName”: “GOOGLE_Pixel2XL_Android_11.0.0_d22ac”,
“platformName”: “Android”,
“automationName”: “uiautomator2”,
“newCommandTimeout”: “600”,
“launchTimeout”: “90000”,
“appPackage”: “com.pcloudy.appiumdemo”,
“appActivity”: “com.ba.mobile.LaunchActivity”,
“pCloudy_EnableVideo” : “true”, # optional
};
Plain text
Copy
The “pCloudy_Username” and “pCloudy_ApiKey” fields can be customized as per the user’s credentials.
Parallel Test
Use the parallel-Test-code to run the parallel test build . Once you have done this, you will need to specify the desired capabilities required to execute the Sample-test.
commonCapabilities: {
browserName: “”,
pCloudy_Username:”Enter-Email”,
pCloudy_ApiKey: “Enter API KEY”,
pCloudy_ApplicationName: “pCloudyAppiumDemo.apk”,
pCloudy_DurationInMinutes: “10”,
platformName: “Android”,
automationName: “uiautomator2”,
newCommandTimeout: “600”,
launchTimeout : “90000”,
appPackage: “com.pcloudy.appiumdemo”,
appActivity: “com.ba.mobile.LaunchActivity”,
pCloudy_EnableVideo : “true”,
},
multiCapabilities: [
{
“pCloudy_DeviceFullName”: “GOOGLE_Pixel7Pro_Android_13.0.0_dbf82”,
},
{
“pCloudy_DeviceFullName”: “MOTOROLA_Edge30Ultra_Android_12.0.0_15b11”,
}
Plain text
Copy
The commonCapabilities field contains properties such as browserName, pCloudy_Username, and pCloudy_ApiKey, which need to be customized as per your credentials. In the multiCapabilities field, you can add as many devices as you require by specifying the pCloudy_DeviceFullName property for each device. This will enable you to execute the Sample-test on multiple device-browser combinations in parallel, leveraging the benefits of pCloudy’s cloud-based testing infrastructure.
Running Tests using Jenkins
Overview
Jenkins is an open-source continuous integration/continuous delivery and deployment (CI/CD) automation software DevOps tool. It is used to implement CI/CD workflows, called pipelines. Users can use Jenkins to execute automation on the pCloudy platform to enable CI/CD.
Pre-requisites
- The user should be registered on the pCloudy platform
- Jenkins installed on the local machine
- Maven and JDK path should be configured in Jenkins
- Maven should be installed on the local machine
- Set the required capabilities in the Appium script
- Copy the project path and keep it handy
Let’s see the Approaches
- Freestyle
- Pipeline
Steps to run jenkins using the Freestyle approach
- Go to Dashboard and Click on New Item
- Enter “project name”, select “Freestyle project” and click “OK”
- Under General scroll down and Go to Build and click on the Add build step
- Select “Execute Windows batch command” for windows or Execute Shell from the drop-down
Note: This needs to be selected depending on OS
- Enter the maven command to run the project and save it
Note: Here user needs to give the path of the project where it is located
E.g. cd C:\Users\Admin\eclipse-workspace\Chapter4-TestNgwithpCloudyonSingleDeviceIOS2
mvn test
Note: Make sure pCloudy capabilities should be set in Runner.java
- Set app capability
**Steps to Set App Capability **- Login to your registered account.
- On the “Start page” navigate to “Capabilities” in the tools section.
- On the Capabilities Configurator Page, you can select OS and the Automation Type.
- Next, you can you can enable different capabilites like Capturing Logs, Enabling local testing or Wildnet, Capturing Video, and capturing Performance Data.
- Next, you can you can enter the Device Details such as region, Device Full name, Manufacturer or Device Version.
- You can select the Device Full Name from the drop down of available devices. However, it is advisable to select the Device Manufacturer and Version instead of the full name of the device to avoid failure of execution due to non-availability of a specific device that you might select in the Device Full name field. Providing a Device Manufacture and Version, gives you a broader reach when your script starts to look for available devices on the platform.
- Once you have selected the Device Details, simply fill in the duration of the execution in mins, selection the App from the drop down, please ensure that you have uploaded the app beforehand on the My Data section. The app package and activity.
- Click on Generate Capabilites, once all the details are filled in.
- The Capabilities will get generated according to the details and specifications that the user mentions. A success notification will appear as well.
- Now you can simply click on the Copy icon to copy the Capabilites and paste it in your appium script and execute the program.
- To startover for a different specification or scenario, simply hit the reset button on the bottom and follow the same steps mentioned above.
- Click on “Build Now” and go to console output to check the status.
- Click on “Console Output”
Steps to run jenkins using the Pipeline approach
Prerequisite:
- Install jq package (This package helps to get the output response)
- jq is like sed for JSON data – you can use it to slice and filter and map and transform structured data with the same ease that sed, awk, grep and friends let you play with text. It is written in portable C.
- Maven should be configured in Jenkins
- User should have a Github account: this is needed incase if the repository is private
Note: This sample project is for reference with respect to the ubuntu machine
Steps
- Create a GitHub account and upload your project
Note
- For the public repository, users don’t need to configure Github in Jenkins
- For a private repository, users need to configure Github in Jenkins
- Login into your Jenkins and click on “New Item”
- Enter the Project name and Select pipeline style.
- If the Project in Github is private, then the user needs to configure Github credentials in Jenkins.
Steps to access Github from Jenkins:-- once the build is created, go to the Jenkins dashboard and click on pipeline syntax to configure GitHub credentials
- In the “sample step” select the “Git” option from the dropdown.
- Enter the repository URL from GitHub.
- Enter the branch name
- Configure GitHub credentials in Jenkins
Note: If Git is not working from the password, generate a git token from GitHub and place it in the password field.
- Click on Generate pipeline script.
Note: Above process is used to clone the Git repository to the Jenkins workspace.
- Go to the Dashboard and click on the created build and select configure
- Write the code to run the pipeline script and click on SAVE
pipeline {
agent any
stages {
stage(‘Clonning Git’) {
steps {
git credentialsId: ”, url: ‘https://github.com/satyamraii/pCloudySample.git’
}
}
stage(‘Authentication’) {
steps {
script {
env.authtoken = sh( script: “curl -u : https://device.pcloudy.com/api/access | jq -r .result.token”,returnStdout: true).trim()
echo “My authotoken: ${env.authtoken}”
}
}
}
stage(‘UploadFile’) {
steps {
sh “curl -X POST -F file= -F source_type=raw -F token=${env.authtoken} -F filter=all https://device.pcloudy.com/api/upload_file”
}
}
stage(‘runtestcases’) {
steps {
dir(env.WORKSPACE){
sh “mvn test”
}
}
}
}
}
Code definition:- There are four stages as mentioned below:-
- Jenkins stage 1:- Git clone-> For example, we have considered pCloudy-sample-project chapter-1: Which is for a single device(Android)
Here is a reference link for the same.
Note: We have updated the capabilities in GitHub as well.
- Jenkins stage 2:- In this stage, we are using the authentication API which is present in content.pcloudy.com
Here the user needs to enter the email and API key to generate the token.
Note: The user can also use the Jq package to help extract the token value from the output
- Jenkins stage 3:- Upload stage-> The user needs to enter the local file path in the API
- Jenkins stage 4 :- Run mvn test.
- Run the build.
Conclusion
CircleCI and Jenkins, each with their unique strengths, cater to different needs. CircleCI stands out with its simplicity and performance, making it an ideal choice for teams looking for a hands-off, managed CI/CD solution. Jenkins, on the other hand, shines with its flexibility and customization capabilities, making it suitable for teams that require unique, complex workflows.
The integration of CircleCI and Jenkins with pCloudy elevates the quality assurance process, facilitating continuous testing in a CI/CD pipeline, and ensuring optimal user experience across various devices and operating systems. The choice between CircleCI and Jenkins will ultimately depend on your team’s specific requirements, infrastructure, and expertise.
August 16, 2023
June 20, 2023
