Installing

Jenkins EPO is a Python3.4+ software configured by environment variables. The package ships a systemd unit reading environment variable from /etc/jenkins-epo.conf file.

The recommended way of deploying Jenkins EPO is through Ansible with bersace.jenkins-epo.

But the first requirement is a Jenkins up and running. jenkins-epo list-plugins lists required Jenkins plugins to run managed jobs. It’s up to Jenkins administrator to install these plugins.

Jenkins must be able to clone repositories with HTTPS. Register a Jenkins credentials for HTTPS clone, and set JOBS_CREDENTIALS according to it.

Next step is to have a GitHub API token. You can create one associated with the GitHub user assigned to Jenkins to clone.

Test your settings like this:

GITHUB_TOKEN=XXX JENKINS_URL=http://jenkins.lan JOBS_CREDENTIALS=github-https jenkins-epo process https://github.com/owner/repo1/tree/master

Then write it to Ansible vars or in /etc/jenkins-epo.conf like this:

set -x
GITHUB_TOKEN=XXX
JENKINS_URL="http://jenkins.lan/"
JOBS_CREDENTIALS=github-https
REPOSITORIES=owner/repo1,owner/repo2
SERVER_URL=http://localhost:2819/
set +x
export REPOSITORIES GITHUB_TOKEN JENKINS_URL JOBS_CREDENTIALS

And reload with systemctl restart jenkins-epo. Watch it with journalctl -fu jenkins-epo !

Setting up WebHook

To increase EPO reactivity, you can use webhooks. EPO listen for webhook on port 2819. There is two webhooks entrypoints.

/simple-webhook

Just pass head URL as head GET param:

curl -X POST http://localhost:2819/simple-webhook?head=https://github.com/owner/repo1/tree/master

At the end of each build, jenkins-yml-runner can notify one URL. EPO tells Jenkins which URL to notify using SERVER_URL. SERVER_URL points to EPO public address, accessible from node executing the build:

HOST=0.0.0.0 PORT=2819 SERVER_URL=http://jenkins.lan:2819 jenkins-epo bot

Watch for the following message at the end of your build log:

+ jenkins-yml-runner notify
Notifying http://jenkins.lan:2819/simple-webhook?head=https://github.com/owner/repo1/tree/master (POST).
Success: b'{"message": "Event processing in progress."}'.
POST BUILD TASK : SUCCESS

Nice! Persist SERVER_URL in /etc/jenkins-epo.conf and you’re done. EPO and Jenkins communicate to speed up the pipeline! Now you can go further!

/github-webhook

If you can open a port to the world, you can tell GitHub to notify EPO of changes on your protected branches or PR. Here is basically how to setup an nginx proxy to serve EPO to GitHub.

  1. Register the domain, get the certificate, a host, etc.

  2. Configure nginx, here is a sample host configuration.

    # This sample config expose only one URL, with SSL. Adapt to your needs.

upstream webhook_handler {
    # Put here the hostname and port where EPO is listening.
    server jenkins.lan:2819;
}

server {
    # Put here the FQDN GitHub will notify.
    server_name jenkins.company.com;

    access_log  /var/log/nginx/epo-webhook.access.log;
    error_log   /var/log/nginx/epo-webhook.error.log;

    listen 443 ssl;

    # Get or generate a certificate. EPO requires a signed certificate.
    ssl_certificate /etc/ssl/certs/epo-webhook.crt;
    ssl_certificate_key /etc/ssl/private/epo-webhook.key;
    ssl_session_timeout 1d;
    ssl_session_cache shared:SSL:50m;
    ssl_protocols TLSv1.2;
    ssl_prefer_server_ciphers on;
    ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256';
    ssl_stapling on;
    ssl_stapling_verify on;

    # HSTS maxage 15768000 seconds = 6 months
    add_header Strict-Transport-Security max-age=15768000;

    real_ip_header   X-Forwarded-For;
    set_real_ip_from 10.0.0.0/8;
    set_real_ip_from 127.0.0.0/8;
    set_real_ip_from 192.168.0.0/16;
    # Add other safe proxy (ngrok, cloudflare, etc.)
    # set_real_ip_from ...

    location /github-webhook {
        allow 10.0.0.0/8;
        allow 127.0.0.0/8;
        allow 192.168.0.0/16;
        # https://help.github.com/articles/github-s-ip-addresses/
        allow 192.30.252.0/22;
        deny  all;

        proxy_redirect      off;
        proxy_set_header    Host                    $http_host;
        proxy_set_header    X-Forwarded-For         $http_x_real_ip;
        proxy_pass          http://webhook_handler;
    }
}

  3. Test it!

    curl -X POST -H 'X-Forwarded-For: 8.8.8.8' https://jenkins.company.com/github-webhook
curl -X POST -H 'X-Forwarded-For: 192.30.252.25' https://jenkins.company.com/github-webhook

  4. Now register EPO in GitHub.

    You need an admin GITHUB_TOKEN. You should use a separate admin token. For example, use a personnal token of yours.

    To increase security, EPO shares a secret with GitHub to sign payload. Save it in /etc/jenkins-epo.conf.

    export GITHUB_SECRET=$(pwgen 64 1)
SERVER_URL=https://jenkins.company.com/ GITHUB_TOKEN=XXX jenkins-epo register

    Jenkins and GitHub can ping different URLs. Just override SERVER_URL with GitHub URL when calling register.

  5. In GitHub web interface, you can test webhook delivery, ping webhook and redeliver a payload.

Now test it for real : push a new commit in a PR and see how fast the jobs are triggered!

Without GitHub webhook

You can still ping EPO with /simple-webhook (e.g. in git hooks, make rule, etc.).

You may want to decrease POLL_INTERVAL. EPO will throttle heads processing to spread GitHub API calls to fit the limit of 5000 calls per hour.

Adding a new repository

EPO can manage multiple repositories! Here are the steps to add a repository to EPO.

  1. Add your bot user to repository’s collaborators with write access.

  2. Ensure your default branch is protected if you want it to be tested!

  3. Add owner/repo2 to REPOSITORIES setting in /etc/jenkins-epo.conf.

  4. Restart EPO:

    systemctl restart jenkins-epo

  5. Watch it with journalctl -fu jenkins-epo, you should see:

    =a892= [INFO    ] Working on https://github.com/owner/repo2/tree/master (a892eb2).
=a892= [WARNING ] No jenkins.yml. Skipping.
=a892= [INFO    ] Processed https://github.com/owner/repo2/tree/master (a892eb2).

  6. Register GitHub webhook for this repository:

    GITHUB_TOKEN=XXX REPOSITORIES=owner/repo2 jenkins-epo register

  7. Create a PR to add jenkins.yml.

Enjoy!