Deploying a YouTube Clone App with DevSecOps and Jenkins Shared Library
In these project, we are cloning the YouTube API With Jenkins Shared Library Using the All the CI/CD And Security Plugins. these project Specially dedicated for Hoe's the Jenkins shared library Works.

Step 1: Launch an Ubuntu 22.04 instance for Jenkins
Log into AWS Console: Sign in to your AWS account.
Launch an Instance:
Choose "EC2" from services. Click "Launch Instance."
Choose an AMI: Select an Ubuntu image.
Choose an Instance Type: Pick "t2.large."
Key Pair: Choose an existing key pair or create a new one.
Configure Security Group:
Create a new security group. Add rules for HTTP, and HTTPS, and open all ports for learning purposes. Add Storage: Allocate at least 20 GB of storage.
[ opening all ports is not recommended for production environments; it's just for educational purposes. ]
Launch Instance: Review and launch the instance.
Access Your Instance: Use SSH to connect to your instance with the private key.

For the installation of the Jenkins follow the official Document.
You will need to go to your AWS EC2 Security Group and open Inbound Port 8080 since Jenkins works on Port 8080.
For unlocking the Jenkins use this command

Jenkins Dashboard
Step2A: Install Docker on the Jenkins machine
Run the below commands to install the docker
After the docker installation, we will create a Sonarqube container (Remember to add 9000 ports in the security group).
Run this command on your EC2 instance to create a SonarQube container:

Now copy the IP address of the ec2 instance

Enter username and password, click on login and change password
SonarQube Dashboard

Step2B: Install Trivy on Jenkins machine
Provide executable permissions and run the shell script

This will install Trivy on our Jenkins machine.

Step3A: Launch an Ubuntu instance for Splunk
Step 1: Launch Instances
Log in to your AWS console or your chosen cloud provider.
Navigate to the EC2 service and launch Ubuntu 22.04 instances. Ensure you select T2.medium as the instance type and allocate 24GB of storage to each instance.
Step 2: Install Splunk
At this point, the first machine is set up with Jenkins. You can now move to the second machine and proceed with the installation of Splunk.
Connect to your second instance using Putty or Mobaxtreme.
To download and install Splunk on your Ubuntu instance use the
wgetcommand, use the following command.

To Depackage the Splunk use the below command
By running this command, you ensure that Splunk Enterprise is configured to start automatically when your Ubuntu system boots, allowing you to seamlessly integrate it into your workflow.
Please note that after running this command, you should follow the on-screen prompts to accept the terms and complete the setup to 100%.
After completing the initial setup and accepting the terms, you'll be prompted to create an admin user.
Administrator Username: Choose a username for the admin account. This should be a unique and secure username.
Administrator Password: Set a strong and secure password for the admin account. It's important to choose a password that combines upper and lower-case letters, numbers, and special characters for enhanced security.
Confirm your password to ensure it matches the one you initially entered.
By creating an administrator username and password, you'll have full access to your Splunk instance, allowing you to configure and manage it effectively.




After Installing set up the Splunk Account- For reference please check official document
Step3B: Install the Splunk app for Jenkins
In Splunk Dashboard
Click on Apps --> Find more apps

Search for Jenkins in the Search bar
You will get the Splunk app for Jenkins and click on install


On the homepage of Splunk, you will see Jenkins has been added

In the Splunk web interface, go to Settings > Data Inputs.

Click on HTTP Event Collector. -> Click on Global Settings
Set All tokens to enabled
Uncheck SSL enable
Use 8088 port and click on save

Now click on New token -> Provide a Name and click on the next

Click Review & Submit

Click Start searching

Now let's copy our token again
In the Splunk web interface, go to Settings > Data Inputs. -> Click on the HTTP event collector
Now copy your token and keep it safe

Add Splunk Plugin in Jenkins
Go to Jenkins dashboard
Click on Manage Jenkins --> Plugins --> Available plugins
Search for Splunk and install it.

Again Click on Manage Jenkins --> System
Search for Splunk
Check to enable
HTTP input host as SPLUNK PUBLIC IP
HTTP token that you generated in Splunk
Jenkins IP and apply.

Now go to Putty or Mobaxtreme and In Splunk machine run this command

Restart Both Splunk and Jenkins
Let's Restart our Splunk machine
Click on Settings --> Server controls

Now restart Jenkins and log in again.

Now go to Splunk and click on the Jenkins app and you will get this output monitoring

Step4A: Integrate Slack for Notifications


Create a Slack account
Open your preferred web browser (e.g., Google Chrome, Mozilla Firefox, Safari, or Microsoft Edge).
In the address bar, type or click on the following URL to access Slack's official website: https://slack.com
On the Slack homepage, you'll find a prominent "SIGN UP WITH GOOGLE" or "Try for Free" button. Click on it


Add the Channel -


Step4B: Install the Jenkins CI app on Slack
Go to Slack and click on your name
Select Settings and Administration
Click on Manage apps

It will open a new tab
Search for Jenkins CI and click on it

Now choose your Slack channel
Click on Add Jenkins CI integration.

You will be redirected to this page

Copy the team subdomain and integration token credential ID for later use.

Team subdomain:
mrbhushanIntegration token credential ID: Create a secret text credential using
LpaDDc2n96KFOLtQ4mDqkxEQas the value
Install Slack Notification Plugin in Jenkins
Go to Jenkins Dashboard
Click on manage Jenkins --> Plugins --> Available plugins
Search for Slack Notification and install

Click on Manage Jenkins --> Credentials --> Global
Select kind as Secret Text
At Secret Section Provide Your Slack integration token credential ID
Id and description are optional and create

Click on Apply and save
manage Jenkins -> add secrete credentials of slack & workspace is slack subdomain

Apply & save.
Step5A: Start Job
Go to Jenkins dashboard and click on New Item.
Provide a name for the Job & click on Pipeline and click on OK.
Step5B: Create a Jenkins shared library in GitHub
Create a new repository in GitHub named Jenkins_shared_library.

Connect to your VS Code
Create a directory named Jenkins-shared-library
Create a Vars directory inside it


Now, Let's Write a Groovy script for our Pipeline
Create a cleanWorkspace.groovy file and add the below code
Create checkoutGit.groovy file and add the below code
Now push them to GitHub using the below commands from vs code


Step5C: Add Jenkins shared library to Jenkins system
Go to Jenkins Dashboard
Click on Manage Jenkins --> system
Search for Global Pipeline Libraries and click on Add

Now Provide a name that we have to call in our pipeline

Step5D: Run Pipeline
Go to Jenkins Dashboard again & select the job and add the below pipeline

Slack Notifications-

Step6: Install Plugins like JDK, Sonarqube Scanner, NodeJs
Step6A: Install Plugin
Goto Manage Jenkins →Plugins → Available Plugins →
Install below plugins
1 → Eclipse Temurin Installer (Install without restart)
2 → SonarQube Scanner (Install without restart)
3 → NodeJs Plugin (Install Without restart)

Step6B: Configure Java and Nodejs in Global Tool Configuration
Goto Manage Jenkins → Tools → Install JDK(17) and NodeJs(16)→ Click on Apply and Save


Step6C: Configure Sonar Server in Manage Jenkins
Grab the Public IP Address of your EC2 Instance, Sonarqube works on Port 9000, so <Public IP>:9000.
Goto your Sonarqube Server. Click on Administration → Security → Users → Click on Tokens and Update Token → Give it a name → and click on Generate Token ->click on update Token


copy Token
Goto Jenkins Dashboard → Manage Jenkins → Credentials → Add Secret Text. It should look like this

Now, go to Dashboard → Manage Jenkins → System and Add like the below image.
Click on Apply and Save.

The Configure System option is used in Jenkins to configure different server
Global Tool Configuration is used to configure different tools that we install using Plugins
We will install a sonar scanner in the tools.

In the Sonarqube Dashboard add a quality gate also
Administration--> Configuration-->Webhooks

Click on Create

Add details

Step6D: Add New stages to the pipeline
Go to vs code and create a file sonarqubeAnalysis.groovy & add the below code and push to Jenkins shared library GitHub Repo.

Create another file for qualityGate.groovy

Create another file for npmInstall.groovy

Push them to the GitHub Jenkins shared library
Add these stages to the pipeline now
Build now.





To see the report, you can go to Sonarqube Server and go to Projects.

Step7: Install OWASP Dependency Check Plugins
GotoDashboard → Manage Jenkins → Plugins → OWASP Dependency-Check. Click on it and install it without restart.

First, we configured the Plugin and next, we had to configure the Tool
Goto Dashboard → Manage Jenkins → Tools →

Click on Apply and Save here.
Create a file for trivyFs.groovy

Push to GitHub
Add the below stages to the Jenkins pipeline

Stage with the Dependency Check steps cannot be directly used inside a shared library.
The main reason is that pipelines loaded from shared libraries have more restrictive script security by default. So the dependencyCheck and dependencyCheckPublisher steps would fail with rejected signature errors.
Build now

You will see that in status, a graph will also be generated and Vulnerabilities.

Step8A: Docker Image Build and Push
We need to install the Docker tool in our system, Goto Dashboard → Manage Plugins → Available plugins → Search for Docker and install these plugins
Docker
Docker Commons
Docker Pipeline
Docker API
docker-build-step
and click on install without restart

Now, goto Dashboard → Manage Jenkins → Tools →

Add DockerHub Username and Password under Global Credentials

Step8B: Create an API key from Rapid API
Open a new tab in the browser and search for rapidapi.com
It will automatically provide your mail and select a mail to create an account

Account is created
Now in the search bar search for YouTube and select YouTube v3


Copy API and use it in the groovy file
docker build --build-arg REACT_APP_RAPID_API_KEY=<API-KEY> -t ${imageName} .
Create a shared library file for dockerBuild.groovy

Create another file for trivyImage.groovy

Push the above files to the GitHub shared library.
Add this stage to your pipeline with parameters
Build now with parameters

Step8C: Run the Docker container
Create a new file runContainer.groovy

Create Another file to remove container removeContainer.groovy

Push them to the Shared library GitHub repo
Add the below stages to the Pipeline

Build with parameters 'create'

It will start the container
Output-

Build with parameters 'delete'

It will stop and remove the Container






Step9A: Kubernetes Setup
Connect your machines to Putty or Mobaxtreme
Take-Two Ubuntu 20.04(t2.medium) instances one for k8s master and the other one for worker.
Install Kubectl on Jenkins machine also.
Step9B: Kubectl is to be installed on Jenkins
Connect your Jenkins machine
Create a shell script file kube.sh
Paste the below commands
Step9C: K8S Master-Slave setup
Part 1 ----------Master Node------------
----------Worker Node------------
Part 2 ------------Both Master & Node ------------
Part 3 --------------- Master ---------------
----------Worker Node------------

Copy the config file to Jenkins master or the local file manager and save it

copy it and save it in documents or another folder save it as secret-file.txt
Note: create a secret-file.txt in your file explorer save the config in it and use this at the kubernetes credential section.
Install Kubernetes Plugin, Once it's installed successfully.

goto manage Jenkins --> manage credentials --> Click on Jenkins global --> add credentials

Step9D: Install Helm & Monitoring K8S using Prometheus and Grafana
On Kubernetes Master install the helm

See the Helm version

Add Prometheus Helm repo
Create Prometheus namespace
Install kube-Prometheus-stack
Below is the command to install kube-Prometheus-stack. The helm repo kube-stack-Prometheus (formerly Prometheus-operator) comes with a Grafana deployment embedded.
Let's check if the Prometheus and Grafana pods are running or not
Now See the services

This confirms that Prometheus and grafana have been installed successfully using Helm.
To make Prometheus and grafana available outside the cluster, use LoadBalancer or NodePort instead of ClusterIP.
Edit Prometheus Service
kubectl edit svc stable-kube-prometheus-sta-prometheus -n prometheus
Edit Grafana Service

Verify if the service is changed to LoadBalancer and also get the Load BalancerPorts.

Access Grafana UI in the browser
Get the external IP from the above screenshot and put it in the browser


Create a Dashboard in Grafana
In Grafana, we can create various kinds of dashboards as per our needs.
How to Create Kubernetes Monitoring Dashboard?
For creating a dashboard to monitor the cluster:
Click the '+' button on the left panel and select ‘Import’.
Enter the 15661 dashboard id under Grafana.com Dashboard.
Click ‘Load’.

Select ‘Prometheus’ as the endpoint under the Prometheus data sources drop-down.

Click ‘Import’.
This will show the monitoring dashboard for all cluster nodes

How to Create Kubernetes Cluster Monitoring Dashboard?
For creating a dashboard to monitor the cluster:
Click the '+' button on the left panel and select ‘Import’.
Enter 3119 dashboard ID under Grafana.com Dashboard.
Click ‘Load’.
Select ‘Prometheus’ as the endpoint under the Prometheus data sources drop-down.
Click ‘Import’.
This will show the monitoring dashboard for all cluster nodes



Create a POD Monitoring Dashboard
For creating a dashboard to monitor the cluster:
Click the '+' button on the left panel and select ‘Import’.
Enter 6417 dashboard ID under Grafana.com Dashboard.
Click ‘Load’.
Select ‘Prometheus’ as the endpoint under the Prometheus data sources drop-down.
Click ‘Import’.



Step9E: K8S Deployment
Let's Create a Shared Jenkins library file for K8s deploy and delete
Name kubeDeploy.groovy

To delete deployment
Name kubeDelete.groovy

Let's push them to GitHub
The final stage of the Pipeline

Build Now with parameters 'create'
It will apply the deployment
stage view

Output -


Build with parameter 'delete'
It will destroy Container and Kubernetes deployment.

Dependacy Check Result -

Slack Notifications

Splunk


Grafhana -





Pipeline
Last updated