Complete guide to set up robust Android CI/CD environment

Image for post
Image for post
by Tomasz Smykowski

From this article you will learn:

  1. How to set up complete Android CI/CD environment
  2. How to set up GitLab runners that execute GitLab pipelines on a computer
  3. How to create GitLab CI/CD for an Angular project
  4. How to boost performance with a private image
  5. How to create own Angular Docker Image
  6. How to upload it to a Nexus registry
  7. How to configure GitLab to use own Docker Image

Lets move on!

Setting up GitLab CI/CD and GitLab Runner

First, we will create GitLab runner. They are used to execute GitLab pipelines on a computer. When a developer will push code to the repository, GitLab will send information to GitLab Runner on a computer to execute pipeline. So, you have to have a computer with GitLab Runner installed and configured to CI/CD to happen.

Creating GitLab account and fetching GitLab Runner registration token

So let’s start setting up GitLab CI/CD:

Create GitLab account. Create a group. Go Settings -> CI/CD. Expand Runners. Copy registration token.

Installing GitLab Runner

Important links:

Download GitLab Runner. Rename downloaded exe file to gitlab-runner.exe and put it into a new folder eg. c:\gitLabRunner. Go to properties of the file and unlock it.

Now call following command to register GitLab Runner in the system:

./gitlab-runner.exe register

After you execute it, you will be asked about several parameters. Below you can find out what you can set up:

Coordinator: https://gitlab.com
Token: paste registration token here obtained in previous step
Description: test
Tags: test
Executor: docker
Default docker image: alpine:latest

Now, install runner. Use elevated cmd (find cmd in start menu, right click -> run as administrator):

gitlab-runner.exe install

If you call it again, and it says gitlab-runner is already running, it means it worked fine.

gitlab-runner.exe start

Now gitlab runner will start automatically with Windows. If you want to read more about the process you can find it here.

Great! You have GitLab and GitLab Runner set up. Now you are ready to start executing pipelines!

Now, we will move on to creating an Android project, we will use to test pipeline, and see if everything works fine.

Setting up Android project and CI/CD script

Important links

Before you have set up a GitLab account, and installed GitLab Runner. Now, you can run pipelines for your project. In order to do it, you need to have a Angular application project, and configure a CI/CD script that tells GitLab Runner how the pipeline should work. Let’s begin!

Create a new project inside a group you created before in GitLab. Clone the repository. Create a .gitlab-ci.yml file. Now we will use boilerplate CI/CD script for Angular. Later you can adjust it to your needs.

Paste this configuration (from here):

image: openjdk:8-jdk

variables:
ANDROID_COMPILE_SDK: "28"
ANDROID_BUILD_TOOLS: "28.0.2"
ANDROID_SDK_TOOLS: "4333796"

before_script:
- apt-get --quiet update --yes
- apt-get --quiet install --yes wget tar unzip lib32stdc++6 lib32z1
- wget --quiet --output-document=android-sdk.zip https://dl.google.com/android/repository/sdk-tools-linux-${ANDROID_SDK_TOOLS}.zip
- unzip -d android-sdk-linux android-sdk.zip
- echo y | android-sdk-linux/tools/bin/sdkmanager "platforms;android-${ANDROID_COMPILE_SDK}" >/dev/null
- echo y | android-sdk-linux/tools/bin/sdkmanager "platform-tools" >/dev/null
- echo y | android-sdk-linux/tools/bin/sdkmanager "build-tools;${ANDROID_BUILD_TOOLS}" >/dev/null
- export ANDROID_HOME=$PWD/android-sdk-linux
- export PATH=$PATH:$PWD/android-sdk-linux/platform-tools/
- chmod +x ./gradlew
# temporarily disable checking for EPIPE error and use yes to accept all licenses
- set +o pipefail
- yes | android-sdk-linux/tools/bin/sdkmanager --licenses
- set -o pipefail

stages:
- build
- test

lintDebug:
stage: build
script:
- ./gradlew -Pci --console=plain :app:lintDebug -PbuildDir=lint

assembleDebug:
stage: build
script:
- ./gradlew assembleDebug
artifacts:
paths:
- app/build/outputs/

debugTests:
stage: test
script:
- ./gradlew -Pci --console=plain :app:testDebug

As a Android application we will use a public simple application. It will make it easier to test the whole process before you switch it to your real application. Download ZIP of Hello World app and put it into your project folder.

Push it to remote repo.

Go to CI/CD. You should see a pipeline running. It is your first CI/CD integration for Android app on Windows with GitLab. Congratulations! If all pipeline points will become green, it means you succeded!

How to download application APK file?

The result of a pipeline in our example is also a APK file you can use to install Android application on your device, or push it to the Android Play Store. Sometimes you would like to send the APK automatically somewhere, but this is not covered by this article. But once you have APK generated you can move further from this point. So since we know APK is generated, how to grab it?

It is possible because of following line:

artifacts:
paths:
- app/build/outputs/

It instructs GitLab Runner to copy files from a given path, and upload it to GitLab. Than you can download it from GitLab interface. You can download files as artifacts in the right panel inside a pipeline ‘assembleDebug’ job step in GitLab.

Making GitLab Docker Android CI/CD faster with own Docker image from Nexus

If you look at the Dockerfile above, it basically downloads and installs Android SDK every time you execute a test. It means each time GitLab Runner will spend processor cycles for the same task. But we are hackers. We can fix that. To acomplish it you need to create a Docker image that already has Android SDK installed and configured.

Than, you can store your Docker Image somewhere, and use it instead of openjdk:8-jdk to gain performance boost.

Now, i will tell you how to set up your computer to create Docker images, how to store the image inside Nexus registry. Nexus registry is one of many places you can store images. It is a private registry where you can put your Docker images. Next, i will show you how to configure GitLab and your CI/CD script file to use your own, prebuilt Docker image.

Installing Docker

Important links

If you have Windows 10 you have to have Insider Preview to executed latest stable version as of 29.07.2020 the new version of Windows 10 is not rolled out everywhere.

But what you can do, is to install Docker Toolbox. it comes with docker inside c:\Program Files\Docker Toolbox. In my case it created user system paths, but not system system paths.

To fix it 1) add system property called:

property: DOCKER_TOOLBOX_INSTALL_PATH
value: C:\Program Files\Docker Toolbox

2) add “C:\Program Files\Docker Toolbox” to system property called “Path” that stores paths to execute things from anywhere.

Also you need to enable Virtualization in the BIOS. Otherwise, opening “Docker Quickstart Terminal” will give you an error:

Error creating machine: Error in driver during machine creation: This computer doesn't have VT-X/AMD-v enabled. Enabling it in the BIOS is mandatory

When you enable Virtualization in BIOS (or UEFI) you can move further. Ps. In my BIOS it was called something like Intel Virtual Technology. If it won’t work double check if you saved changed in BIOS.

Now you can call “Docker Quickstart Terminal” — a shortcut made by Docker on your desktop. If you don’t have it, call, go to c:\program files\docker toolbox and call:

“C:\Program Files\Git\bin\bash.exe” — login -i “C:\Program Files\Docker Toolbox\start.sh”

It will open Docker console.

Creating docker image

Important links

Next you can create a docker image:

To do it we need to build own image:

mkdir android-docker-image

Create Dockerfile. Paste (from here):

FROM gradle:4.10.0-jdk8
USER root
ENV SDK_URL="https://dl.google.com/android/repository/sdk-tools-linux-3859397.zip" \
ANDROID_HOME="/usr/local/android-sdk" \
ANDROID_VERSION=28 \
ANDROID_BUILD_TOOLS_VERSION=27.0.3
# Download Android SDK
RUN mkdir "$ANDROID_HOME" .android \
&& cd "$ANDROID_HOME" \
&& curl -o sdk.zip $SDK_URL \
&& unzip sdk.zip \
&& rm sdk.zip \
&& mkdir "$ANDROID_HOME/licenses" || true \
&& echo "24333f8a63b6825ea9c5514f83c2829b004d1fee" > "$ANDROID_HOME/licenses/android-sdk-license"
# && yes | $ANDROID_HOME/tools/bin/sdkmanager --licenses
# Install Android Build Tool and Libraries
RUN $ANDROID_HOME/tools/bin/sdkmanager --update
RUN $ANDROID_HOME/tools/bin/sdkmanager "build-tools;${ANDROID_BUILD_TOOLS_VERSION}" \
"platforms;android-${ANDROID_VERSION}" \
"platform-tools"
# Install Build Essentials
RUN apt-get update && apt-get install build-essential -y && apt-get install file -y && apt-get install apt-utils -y

Call:

docker build -t android-build:android-gradle .

(notice the dot at the end!)

If it will work good you will see a nice build progress:

Image for post
Image for post

Make a tea, it will take time…

After it finishes you can call:

docker images -a

To see your shiny Docker Image. 1

Congratulations. Now you have your own Docker image that is tailored to speed up your CI/CD process. Developers will be happy!

Speeding up CI/CD with own Docker images repository

Important links

Most of the time it is nice to have the image stored somewhere. You can store it, for example, in Docker Hub. In the awesome article by Andrés Sandoval you can read how to publish image to DockerHub.

But from performance point of view, it is nice to have it in the same network as your GitLab Runner. To acomplish it, you need to have your own Docker Images repository set up. One of such systems available is Nexus. Following part of the article assumes you have already set up Nexus, or someone in your organisation set it up before. If not, please create one before going further.

Saving Docker Image to a Nexus repository

Here i will cover how to publish Docker Image you created earlier to Nexus repository. First you tag your image:

docker tag android-build:android-gradle <nexus-url>:<nexus-port>/<repository-path>:<repository-tag>

As you can see it is rather complicated. So let’s say we want to store our image on a nexus server that url is dockerrepo.bestsoftwarehouse.com (you have to find out this one, as well as the port being used), accessed by default port number, under “ourdockers/android/test” version 1. Than our command will look like this:

docker tag android-build:android-gradle dockerrepo.bestsoftwarehouse.com/ourdockers/android/test:1

You can see the tag being created with:

docker images -a

The new tag indicates where it should be stored. Now we have to login into the Nexus repository. Beware to call it before pushing docker image. Otherwise you will face ‘no basic auth credentials’ error. So, let’s log in to the Nexus repository:

docker login <nexus-url>:<nexus-port>

In our example it would be:

docker login dockerrepo.bestsoftwarehouse.com

Once we are in, we can push the repo with:

docker push <nexus-url>:<nexus-port>/<repository-path>:<repository-tag>

In our example:

docker push dockerrepo.bestsoftwarehouse.com/ourdockers/android/test:1

If you receive “unauthorized: access to the requested resource is not authorized” message it means Nexus administrator has to give you privilages to push images against the Nexus repository.

If it will succeed, you will see pushing progress and success message at the end. The image is most likely to be > 1 GB, so it will take a lots of time. Also, the process may be broken due to TLS handshake timeout. If it happens you can write:

docker push dockerrepo.bestsoftwarehouse.com/ourdockers/android/test:1 && docker push dockerrepo.bestsoftwarehouse.com/ourdockers/android/test:1 && docker push dockerrepo.bestsoftwarehouse.com/ourdockers/android/test:1 && docker push dockerrepo.bestsoftwarehouse.com/ourdockers/android/test:1 && docker push dockerrepo.bestsoftwarehouse.com/ourdockers/android/test:1

It will resume uploading image when last process will fail to complete.

Interestingly enough the image is transferred via layers. So if a layer will be uploaded in one batch, it does not need to be uploaded again, even if the command will fail at the end to upload whole image.

After you have uploaded the image you can check with Nexus Repository Manager web-UI if the image is available. You should have been given the URL of the repository manager from an admin.

Congratulations! Your custom Docker image is available from private Nexus repository. Now, we need to change settings of our CI/CD to use the new Docker image instead of the standard one.

Setting GitLab to get private Docker image from Nexus

Important links

Now we have an image not in the public Docker repository, but our custom image in a Nexus private repository. In order to use it we need to configure GitLab properly.

First you need to obtain user credentials for Nexus that allows only to pull images. Because credentials you used before were able to push images also. So it is safe to have other user with less privileges for GitLab.

When you have it, you need to generate auth code. It is easy when you have docker installed. All you have to do is to log in to your registry:

docker login <nexus-url>:<nexus-port> --username <username> --password <password>

And extract auth code from:

C:\Users\<your user name>\.docker\config.json

You can find the auth code inside the file above. If not, follow this alternative way of generating auth from login and password: instruction.

Now, when we have auth code, login to your GitLab, go to your project -> settings -> CI\CD -> Variables and add variable called DOCKER_AUTH_CONFIG with this content:

{
"auths": {
"<nexus-url>:<nexus-port>": {
"auth": "<auth>"
}
}
}

If your nexus is available on default port on dockerrepo.bestsoftwarehouse.com it will look like this:

{
"auths": {
"dockerrepo.bestsoftwarehouse.com": {
"auth": "<auth>"
}
}
}

We are done with GitLab. GitLab will now have auth code, pass it to GitLab Runner. GitLab Runner will use auth code to authorize with Nexus repository.

It is the time to change our CI/CD script to fetch Docker image from Nexus rather than from Docker Hub.

As you can recall earlier in the article we have set up Hello World application. Also, we have created CI/CD script file called ‘.gitlab-ci.yml’. The boilerplate script looked like this:

image: openjdk:8-jdkvariables:
DOCKER_DRIVER: overlay2
ANDROID_COMPILE_SDK: "28"
ANDROID_BUILD_TOOLS: "28.0.2"
ANDROID_SDK_TOOLS: "4333796"
before_script:
- apt-get --quiet update --yes
- apt-get --quiet install --yes wget tar unzip lib32stdc++6 lib32z1
- wget --quiet --output-document=android-sdk.zip https://dl.google.com/android/repository/sdk-tools-linux-${ANDROID_SDK_TOOLS}.zip
- unzip -d android-sdk-linux android-sdk.zip
- echo y | android-sdk-linux/tools/bin/sdkmanager "platforms;android-${ANDROID_COMPILE_SDK}" >/dev/null
- echo y | android-sdk-linux/tools/bin/sdkmanager "platform-tools" >/dev/null
- echo y | android-sdk-linux/tools/bin/sdkmanager "build-tools;${ANDROID_BUILD_TOOLS}" >/dev/null
- export ANDROID_HOME=$PWD/android-sdk-linux
- export PATH=$PATH:$PWD/android-sdk-linux/platform-tools/
- chmod +x ./gradlew
# temporarily disable checking for EPIPE error and use yes to accept all licenses
- set +o pipefail
- yes | android-sdk-linux/tools/bin/sdkmanager --licenses
- set -o pipefail
stages:
- build
- test
lintDebug:
stage: build
script:
- ./gradlew -Pci --console=plain :app:lintDebug -PbuildDir=lint
assembleDebug:
stage: build
script:
- ./gradlew assembleDebug
artifacts:
paths:
- app/build/outputs/apk/debug
#- app/build/outputs/
debugTests:
stage: test
script:
- ./gradlew -Pci --console=plain :app:testDebug

Now, since our custom Docker image already has Android SDK we can remove almost half of the script. GitHub Runner won’t need to download and install Android SDK each time pipeline is executed! It is a great performance boost!

Below you can see how the content of ‘.gitlab-ci.yml’ should now look like. Please notice we need to set chmod +x ./gradlew and also, remember to replace image path to point to your Docker image location. It is rather easy because it should look exactly the same like the URL you provided earlier to ‘docker push’ command:

image: <nexus-url>:<nexus-port>/<repository-path>:<repository-tag>before_script:
- chmod +x ./gradlew
stages:
- build
- test
lintDebug:
stage: build
script:
- ./gradlew -Pci --console=plain :app:lintDebug -PbuildDir=lint
assembleDebug:
stage: build
script:
- ./gradlew assembleDebug
artifacts:
paths:
- app/build/outputs/apk/debug
#- app/build/outputs/
debugTests:
stage: test
script:
- ./gradlew -Pci --console=plain :app:testDebug

Commit and push changes to GitLab repository. If you work on master branch pipeline will start immediately.

If everything worked fine, pipeline should start. You will notice how fast it executes! And soon, all pipeline steps will become green. It will indicate you have successfully set up your super-fast CI/CD with GitLab and Nexus for an Android project.

Good work!

What’s next?

In the next article I will cover how to configure ‘.gitlab-ci.yml’ for some common CI/CD scenarios. Follow me, if you want to be notified about it!

Writing this article took me a lots of time, but was also rewarding to be able to write down a simplified process of setting up everything! I hope you enjoyed it too! Please consider clapping for the article if you liked it!

Thanks & Have a nice day!

Written by

Senior software development consultant. Programming for 20 years. TOP 2% of StackOverflow users. 2 million views on Quora. Currently Angular, TypeScript etc

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store