Setting up Teamcity as CI CD on self-hosted server with reverse proxy (nginx)

Date [ 21 Feb 20 16:20 +0400 ] Categories [ tech ] Tags [ ci-cd , self-host , teamcity ]

As part of automating things, I have setup my CI-CD system using Jetbrains Teamcity on my self-hosted server. As the first step I am setting the CD-CI piple for my blog. I am documenting my steps for future reference.

Why Teamcity?

Mostly because it was the first one that worked as I wanted right on the initial try, the concepts matched my requirements of what I wanted to do with a CD server. It also had docker images readily available for both the server and the agent and putting up a customized docker-compose file was easy. I was also able to setup Teamcity behind an Nginx reverse proxy which already had the SSL ready and running.

I am also looking into Drone and will document my experience at a later stage.

Overview

Setup involves a cloud server as both the CI-CD system and the development environment to test the changes before I push it to the final hosting location.

Below is my envisioned flow of the changes:

CI-CD for blog

Server component setup

The setup consist of the following main components:

Nginx - Reverse proxy server
Docker with docker-compose
- Jetbrains Teamcity Server (jetbrains/teamcity-server)
- Jetbrains Teamcity Agent (jetbrains/teamcity-agent)

Customization

The setup was customized so that the in-built webserver of the build server is not exposed to the public IP and is accessed through /teamcity/ path of the domain name pointing to the cloud server.

This is accomplished by

Create a new Location block in Nginx named `/teamcity/` and configure it for reverse proxying

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
map $http_upgrade $connection_upgrade { # WebSocket support
    default upgrade;
    '' '';
}

server {
   listen 80;
   listen [::]:80;

   server_name cicd.example.com; # domain pointing to your server
   return 301 https://$host$request_uri;
}

server {
   listen [::]:443 ssl ipv6only=on; # managed by Certbot
   listen 443 ssl; # managed by Certbot

   server_name cicd.example.com; # domain pointing to your server

  ################################################
  # your regular configuration settings go here...
  ################################################

   location /teamcity/ {
      proxy_read_timeout     1200;
      proxy_connect_timeout  240;
      client_max_body_size   0;    # maximum size of an HTTP request. 0 allows uploading large artifacts to TeamCity

      proxy_pass http://localhost:1056; # full internal address
      proxy_http_version  1.1;
      proxy_set_header    Host $server_name:$server_port;
      proxy_set_header    X-Forwarded-Host $http_host;    # necessary for proper absolute redirects and TeamCity CSRF check
      proxy_set_header    X-Forwarded-Proto $scheme;
      proxy_set_header    X-Forwarded-For $remote_addr;
      proxy_set_header    Upgrade $http_upgrade; # WebSocket support
      proxy_set_header    Connection $connection_upgrade; # WebSocket support
   }
}

Customizing `/opt/teamcity/conf/server.xml` file to accept connection from the reverse proxy

Make a copy of the existing server.xml file and change the Connector node to accept reverse proxy details.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
<Connector port="8111" protocol="org.apache.coyote.http11.Http11NioProtocol"
  connectionTimeout="60000"
  useBodyEncodingForURI="true"
  socket.txBufSize="64000"
  socket.rxBufSize="64000"
  tcpNoDelay="1"
  proxyName="cicd.example.com"
  proxyPort="443"
  secure="true"
  scheme="https"
  />

Another approach is to use the RemoteIpValue configuration as mentioned in the Teamcity documentation. This configuration did not work out for me and I went with the Connector based one.

Customize the docker-compose file to handle the port mapping and context as needed

Change the docker-compose.yml file to include the following configs:

Environment variable TEAMCITY_CONTEXT with the value of your nginx Location block path
Map port to loopback IP so that Teamcity server is not listening on the public IP
Map the volumes, especially the customized server.xml file
Agent: Set the SERVER_URL to reflect the context
Agent: Map the required volume to any folder you want to publish (in my case /var/www:/var/www)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
version: '3'

services:
   server:
      image: jetbrains/teamcity-server
      environment:
         - TEAMCITY_CONTEXT=/teamcity/
      ports:
         - 127.0.0.1:8111:8111
      volumes:
         - /home/docker/teamcity/server/data:/data/teamcity_server/datadir
         - /home/docker/teamcity/server/logs:/opt/teamcity/logs
         - /home/docker/teamcity/server/conf/server.xml:/opt/teamcity/conf/server.xml
   agent:
      image: jetbrains/teamcity-agent
      environment:
         - SERVER_URL=http://server:8111/teamcity/
         - AGENT_NAME=build-agent-01
      volumes:
         - /home/docker/teamcity/agent/conf:/data/teamcity_agent/conf
         - /home/docker/teamcity/agent/work:/opt/buildagent/work
         - /home/docker/teamcity/agent/system:/opt/buildagent/system
         - /var/www:/var/www
      depends_on:
         - server

Finally configure the server url configuration.