Private EKS API Endpoint behind OpenVPN

AWS offers a managed Kubernetes solution called Elastic Kubernetes Service (EKS). When an EKS cluster is spun up the Kubernetes API is by default accessible by the public. However, this might be something that your company does not approve of due to security reasons, they might want to limit Kubernetes API access only to private networks. In that case you might want to bring up a service as OpenVPN and route private traffic through it. That would allow you to access the Kubernetes API through a private endpoint using OpenVPN. In this blog post we'll use Terraform to provision our infrastructure required for a private EKS cluster and we'll use OpenVPN Access Server as our VPN solution.

Creating an EKS Cluster

We'll be using the EKS Terraform module to deploy our cluster and the VPC Terraform module to deploy our networking stack. We'll also create two separate VPCs, one exclusively used for the Kubernetes cluster, the other one called Ops where we store common shared resources as the VPN. This allows you to peer the Ops VPC with every other VPC you'd like and use a single VPN solution accross all VPCs.

Creating the VPC

We'll be using the VPC Terraform module to create our VPC. The VPC has nothing specific about it, it is using the 10.11.0.0/16 with 3 private and public subnets. We also add common Kubernetes tags.

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "3.13.0"

  name                 = "staging"
  cidr                 = "10.11.0.0/16"
  azs                  = data.aws_availability_zones.available.names
  private_subnets      = ["10.11.0.0/19", "10.11.32.0/19", "10.11.64.0/19"]
  public_subnets       = ["10.11.96.0/19", "10.11.128.0/19", "10.11.160.0/19"]
  enable_nat_gateway   = true
  enable_dns_hostnames = true

  tags = {
    "kubernetes.io/cluster/${local.cluster_name}" = "shared"
  }

  public_subnet_tags = {
    "kubernetes.io/cluster/${local.cluster_name}" = "shared"
    "kubernetes.io/role/elb"                      = "1"
  }

  private_subnet_tags = {
    "kubernetes.io/cluster/${local.cluster_name}" = "shared"
    "kubernetes.io/role/internal-elb"             = "1"
  }
}

Creating the Cluster

We'll be using the EKS Terraform module to create our EKS cluster. I'll not include the full config specification, only the vpc, subnets and security groups configurations. We'll set the cluster_endpoint_private_access to true and the cluster_endpoint_public_access to false. This disables public access and allows private access. Also, we'll use the vpc and subnets created in the previous step. The security groups will define two additional rules using the cluster_security_group_additional_rules parameter. The rules will allow ingress & egress to and from the Ops VPC(CIDR 10.10.0.0/16) where the OpenVPN instance will be deployed. We'll allow all traffic with all protocols(for use cases as for example if we are using Kubernetes NodePorts, needing SSH access etc.), however you could limit this if you'd like. The below config should enable private access and allow private access from the Ops VPC.

module "eks" {
  source  = "terraform-aws-modules/eks/aws"
  version = "18.20.5"

  cluster_name    = local.cluster_name
  cluster_version = "1.22"

  cluster_endpoint_private_access = true
  cluster_endpoint_public_access  = false

  cluster_security_group_additional_rules = {
    ops_private_access_egress = {
      description = "Ops Private Egress"
      protocol    = "-1"
      from_port   = 0
      to_port     = 0
      type        = "egress"
      cidr_blocks = ["10.10.0.0/16"]
    }
    ops_private_access_ingress = {
      description = "Ops Private Ingress"
      protocol    = "-1"
      from_port   = 0
      to_port     = 0
      type        = "ingress"
      cidr_blocks = ["10.10.0.0/16"]
    }
  }

  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.vpc.private_subnets
}

Deploying the OpenVPN Access Server

We'll be using a community AMI to deploy the OpenVPN Access Server and the VPC Terraform module to deploy our networking stack for the Ops VPC.

Creating the VPC

The Ops VPC where OpenVPN will be deployed will have the CIDR 10.10.0.0/16, this is also the CIDR which was whitelisted when adding the additional security groups for the EKS cluster.

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "3.13.0"

  name                 = local.env
  cidr                 = "10.10.0.0/16"
  azs                  = data.aws_availability_zones.available.names
  private_subnets      = ["10.10.0.0/19", "10.10.32.0/19", "10.10.64.0/19"]
  public_subnets       = ["10.10.96.0/19", "10.10.128.0/19", "10.10.160.0/19"]
  enable_nat_gateway   = true
  single_nat_gateway   = true
  enable_dns_hostnames = true
}

We'll be using the vpc-peering module for AWS to peer our Kubernetes cluster VPC and our Ops VPC.

module "vpc_peering_staging" {
  source  = "cloudposse/vpc-peering/aws"
  version = "0.9.2"

  name             = "${local.env}-staging"
  requestor_vpc_id = module.vpc.vpc_id
  acceptor_vpc_id  = "<The EKS clusters VPC id>"
}

Deploying the OpenVPN Access Server

Now that we have our EKS cluster up and running we'll need to deploy the OpenVPN Access Server to reach the Kubernetes API endpoint. We'll be using an AWS Community AMI for the OpenVPN Access Server. To use the AMI you'll need to accept the terms of the AWS marketplace for that AMI. To do so you'll go to the marketplace page for OpenVPN and subscribe. This will not cost anything, the AMI is free up to two active connections, if you need more you'll need to purchase an OpenVPN license. We'll need the following AWS resources:

A SSH key-pair to access the instance.
A security group that permits ingress network for the ports 443(https web UI, tcp client connections), 80(http redirect web UI), 1194(udp client connections), 22(ssh connectivity).
An EC2 instance.
Optional: A DNS record and an Elastic IP

First we'll start by creating the required resources, below is configurations for the key pair, security group and EC2 instance. We are using the ec2-instance module to create the EC2 instance.

module "ops_key_pair" {
  source  = "terraform-aws-modules/key-pair/aws"
  version = "1.0.1"

  key_name   = "ops"
  public_key = "<my public key>"
}

resource "aws_security_group" "openvpn" {
  name        = "openvpn"
  vpc_id      = module.vpc.vpc_id
  description = "OpenVPN security group"

  tags = {
    Name = "OpenVPN"
  }

  ingress {
    protocol    = -1
    from_port   = 0
    to_port     = 0
    cidr_blocks = ["10.10.0.0/16"]
  }

  # For OpenVPN Client Web Server & Admin Web UI
  ingress {
    protocol    = "tcp"
    from_port   = 443
    to_port     = 443
    cidr_blocks = ["0.0.0.0/0"]
  }

  # For OpenVPN Client Web Server & Admin Web UI
  ingress {
    protocol    = "tcp"
    from_port   = 80
    to_port     = 80
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    protocol    = "udp"
    from_port   = 1194
    to_port     = 1194
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    protocol    = "tcp"
    from_port   = 22
    to_port     = 22
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    protocol    = -1
    from_port   = 0
    to_port     = 0
    cidr_blocks = ["0.0.0.0/0"]
  }
}

module "openvpn_ec2_instance" {
  source  = "terraform-aws-modules/ec2-instance/aws"
  version = "3.5.0"

  name = "OpenVPN"

  ami                    = "ami-0e1415fedc1664f51" // Community edition OpenVPN 2.8.5
  instance_type          = "t2.small"
  key_name               = module.ops_key_pair.key_pair_key_name
  vpc_security_group_ids = [aws_security_group.openvpn.id]
  subnet_id              = module.vpc.public_subnets[0]

  tags = {
    Terraform   = "true"
    Environment = "ops"
  }
}

We'll also add an elastic IP and a dns record to the EC2 instance. This is optional.

resource "aws_eip" "openvpn" {
  instance = module.openvpn_ec2_instance.id
  vpc      = true
}

resource "aws_route53_record" "openvpn" {
  zone_id = data.aws_route53_zone.main.zone_id
  name    = "openvpn.${local.root_domain}"
  type    = "A"
  ttl     = "300"
  records = [aws_eip.openvpn.public_ip]
}

Configuring OpenVPN

Now that we have all infrastructure in place we'll configure OpenVPN. Since we are using a prebuilt AMI, the configuration is quite simple. SSH into your instance using the key pair you provided when creating the EC2 instance. After connecting a script should automatically run where you configure OpenVPN, just choosing the default values should be sufficient. We'll also create a password for the openvpn user with the command passwd openvpn. After the setup you should be able to access the IP of the instance using the browser and you should reach the OpenVPN login page (there should be a SSL warning, but bypass it for now). If you go then to the /admin path you should be able to login with openvpn user.

We'll need to add the CIDR range for the Kubernetes cluster VPC to the OpenVPN routing, you'll do this by going to Configuration -> VPN Settings -> in the routing section add the CIDR for the Kubernetes VPC which in our case is 10.11.0.0/16. This will ensure that any IP accessed in that private range will be routed through the OpenVPN instance. If we use dig to resolve our EKS API endpoint it will resolve to an IP in that range. You can use dig to test it dig <my API server DNS>.eu-west-1.eks.amazonaws.com should return a result similar to this:

;; ANSWER SECTION:
<my-cluster>.eu-west-1.eks.amazonaws.com. 60 IN A 10.11.54.91
<my-cluster>.eu-west-1.eks.amazonaws.com. 60 IN A 10.11.75.122

Grab a OpenVPN client configuration at the root(https://openvpn.my-domain.com/) page and use either the GUI application or CLI to connect to the VPN and you should be able to reach your private EKS endpoint.

SSL Certificates using Certbot

Lastly, we'll add a SSL certificate so that the web server terminates TLS, this is an optional step.

First we'll set the hostname for our OpenVPN instance by going to:

Configuration -> Network Settings -> set the hostname to your DNS record.

Now SSH into the server and run sudo certbot certonly --standalone, this should generate certificates for your domain and you can find the certificates at:

cat /etc/letsencrypt/live/<domain>/cert.pem
cat /etc/letsencrypt/live/<domain>/chain.pem
cat /etc/letsencrypt/live/<domain>/privkey.pem

Add the SSL certificates to the server by going to Configuration -> Web Server -> upload all three files and validate the configuration.

Conclusion

That should be all! Now you can start adding additional users and they will access the Kubernetes API through a private endpoint. On their end nothing will be needed to be changed, if you are changing your EKS Kubernetes cluster from public to private access the DNS will resolve to a private IP instead of public - nothing else changes. They will just need to use the VPN.