Skip to main content

Running Jenkins inside Docker

Whether you are a beginner to Jenkins and want to have your setup done quickly or you want to harness the full power of containers by running Jenkins in Docker, this simple step-by-step tutorial is for you.

Running Jenkins in Docker

We will pull Jenkins Docker image from Docker repository:

docker pull jenkins

Using below command we will map ‘jenkins’ directory in my working directory ($PWD/jenkins) to /var/jenkins_home directory in container and map port 49001 on host to 8080 on container.

docker run -d -p 49001:8080 -v $PWD/jenkins:/var/jenkins_home:z -t jenkins

Now to run jenkins, just give http://localhost:49001/. You can proceed with the setup and management of Jenkins now.

Your entire Jenkins setup has been created in the 'jenkins' directory which you mapped earlier. Go ahead and take a look! 

Packaging app in Docker and pushing to repository

Sometimes you would want to package your app as a Docker image and publish to the Docker Hub (or any other container registry of your choice) as the final step of your build process. If so, follow this step.

We will use CloudBees Docker Build and Publish Plugin. Go ahead and install it using the Manage Jenkins -> Manage Plugins -> Available tab in your Jenkins UI.

To publish to Docker, our jenkins host machine needs to have docker client installed. For this we need to downloaded a different image which is based on Jenkins and also has docker client.

You can stop the docker container which you started in previous step.

To create this new image we will use following dockerfile:

FROM jenkins
USER root

RUN mkdir -p /tmp/download && \
 curl -L https://get.docker.com/builds/Linux/x86_64/docker-1.13.1.tgz | tar -xz -C /tmp/download && \
 rm -rf /tmp/download/docker/dockerd && \
 mv /tmp/download/docker/docker* /usr/local/bin/ && \
 rm -rf /tmp/download && \
 groupadd -g 999 docker && \
 usermod -aG staff,docker jenkins

user jenkins


We create an image named ‘jenkins-docker’ using the above dockerfile with following command. You can change the image name to whatever you want. The '.' implies that the dockerfile is present in this directory. If your dockerfile is in some other directory please give that path.

docker build -t jenkins-docker .

We now run a container with this image:

sudo docker run -d -p 49001:8080 -v $PWD/jenkins:/var/jenkins_home:z -v /var/run/docker.sock:/var/run/docker.sock -t jenkins-docker

In this command we are mapping the docker socket for docker on our machine with the docker socket in the image.

Once this is done, we add ‘Docker Build and Publish’ step in the build steps. We give the repository name (of Docker Hub) to which we want to publish, and its credentials. Now when the build runs, it packages the app to a Docker image and publishes to Docker Hub.

Jenkins Job DSL

DSL: Domain Specific Language

Writing a Jenkins job in Groovy instead of doing it from UI. Writing a simple job of getting a Node JS app from github and doing ‘npm install’ using DSL:

job(‘NodeJS App’) {
scm {
git(‘git://github.com/') { node ->
node / gitConfigName(‘UserName’)
node / gitConfigEmail(‘usermail’)
}
}

triggers {
scm(‘H/5 * * * *’) //How many times to run? Every 5 mins
}

wrappers {
nodejs(‘nodejs’) //name of NodeJS install in Jenkins
}

steps {
shell(“npm install”)
}
}

To setup job DSL, we create a new Freestyle project and add a new build step ‘Process Job DSLs’. There either we can copy the script or if we have specified any repo in Source Code Management, give the path of the Job DSL groovy script.

If we run the build now, it will fail with error: ‘script not yet approved for use’. To approve, go to ‘In-process Script Approval’ and approve the script. Once this build succeeds, we will see a new Node JS project created (as we specified in the DSL script)

Jenkins Pipeline DSL

For creating a pipeline which pulls code from SCM, runs tests and pushes to Docker, we have forked a project ‘docker-demo’ and altered its Jenkinsfile:

node {
   def commit_id
   stage('Preparation') {
     checkout scm
     sh "git rev-parse --short HEAD > .git/commit-id"                        
     commit_id = readFile('.git/commit-id').trim()
   }
   stage('test') {
     nodejs(nodeJSInstallationName: 'nodejs') {
       sh 'npm install --only=dev'
       sh 'npm test'
     }
   }
   stage('docker build/push') {
     docker.withRegistry('https://index.docker.io/v1/', 'dockerhub') {
       def app = docker.build("waterfox83/docker-nodejs-demo:${commit_id}", '.').push()
     }
   }
}

Creating a new ‘Pipeline’ type project with this Jenkinsfile, creates a new project with three stages and pushes the build to docker.

Running Build/Test in Docker Containers

If you want that build/test stages of your build pipeline are run in their own containers, you can do that as well. In this example, there is a stage named 'test with a DB' which requires mySql to be running. We will change our dockerfile so that a new container with mySql image is created and test is run within that container.

It is possible to run everything in containers as part of build pipeline. In below jenkinsfile, we have modified the file from previous example and are running everything in containers.

   stage('test') {
     def myTestContainer = docker.image('node:4.6')
     myTestContainer.pull()
     myTestContainer.inside {
       sh 'npm install --only=dev'
       sh 'npm test'
     }
   }

   stage('test with a DB') {
     def mysql = docker.image('mysql').run("-e MYSQL_ALLOW_EMPTY_PASSWORD=yes") 
     def myTestContainer = docker.image('node:4.6')
     myTestContainer.pull()
     myTestContainer.inside("--link ${mysql.id}:mysql") { // using linking, mysql will be available at host: mysql, port: 3306
          sh 'npm install --only=dev' 
          sh 'npm test'                     
     }                                   
     mysql.stop()
   }                                     


I hope this post has helped you in starting with running Jenkins on Docker.  If you have any questions/comments please post them in comments. Happy learning!

Comments

Popular posts from this blog

How to upload to Google Cloud Storage buckets using CURL

Signed URLs are pretty nifty feature given by Google Cloud Platform to let anyone access your cloud storage (bucket or any file in the bucket) without need to sign in. Official documentation gives step by step details as to how to read/write to the bucket using gsutil or through a program. This article will tell you how to upload a file to the bucket using curl so that any client which doesn't have cloud SDK installed can do this using a simple script. This command creates a signed PUT URL for your bucket. gsutil signurl -c 'text/plain' -m PUT serviceAccount.json gs://test_bucket_location Here is my URL: https://storage.googleapis.com/test_sl?GoogleAccessId=my-project-id@appspot.gserviceaccount.com&Expires=1490266627&Signature=UfKBNHWtjLKSBEcUQUKDeQtSQV6YCleE9hGG%2BCxVEjDOmkDxwkC%2BPtEg63pjDBHyKhVOnhspP1%2FAVSr%2B%2Fty8Ps7MSQ0lM2YHkbPeqjTiUcAfsbdcuXUMbe3p8FysRUFMe2dSikehBJWtbYtjb%2BNCw3L09c7fLFyAoJafIcnoIz7iJGP%2Br6gAUkSnZXgbVjr6wjN%2FIaudXIqA

Running Apache Beam pipeline using Spark Runner on a local standalone Spark Cluster

The best thing about Apache Beam ( B atch + Str eam ) is that multiple runners can be plugged in and same pipeline can be run using Spark, Flink or Google Cloud Dataflow. If you are a beginner like me and want to run a simple pipeline using Spark Runner then whole setup may be tad daunting. Start with Beam's WordCount examples  which help you quickstart with running pipelines using different types of runners. There are code snippets for running the same pipeline using different types of runners but here the code is running on your local system using Spark libraries which is good for testing and debugging pipeline. If you want to run the pipeline on a Spark cluster you need to do a little more work! Let's start by setting up a simple standalone single-node cluster on our local machine. Extending the cluster is as easy as running a command on another machine, which you want to add to cluster. Start with the obvious: install spark on your machine! (Remember to have Java a

java.lang.IllegalArgumentException: Malformed \uxxxx encoding

I was getting this exception during build while running ant. Googling didn't help much and I was flummoxed because the same code was running fine till now. My code reads a text file and does some operations on the basis of values read. It was only when I saw the text files I understood the error. I had copied the text in wordpad and saved it as .txt file. Wordpad had put lot of formatting information before and after the content. Also there was "\par" after every line, which was giving this error. So moral of the story: if you get this exception check your properties file (or any other file that your code might be reading.)