Stop exposing port 22 to the world. It’s time to rework your remote access methods

This post is aimed at showing how remote console-based access to cloud-resources can be achieved without exposing port 22 to the world, and – in fact – without any kind of internet access provided to your virtual resources.

This post is based on AWS technologies, which I’m most familiar with, but similar features are being rolled out by the other Cloud Providers.

The idea is to deploy a ‘bastion’ host in a private subnet on AWS and use that as an SSH proxy to access any other hosts in the same VPC and in any other private subnet, without any requirement to expose any port in any of your Security Groups or NACLs to the Internet.

Clearly, this is not meant to be a production-ready deployment, but just serve as inspiration on how things can be done differently.

The infrastructure

We have:

  1. a private subnet where the Bastion host is deployed
  2. another private subnet(s) where target hosts are deployed
  3. a set of VPC Endpoints (Interface-type) to allow the AWS System Manager to reach and be reached by the Bastion SG ssm-agent on port 443
  4. three separate security groups regulating inbound and outbound access to/from the various groups of hosts/interfaces in the infrastructure.
ssm-bastion

How it all works

AWS System Manager

AWS SSM (System Manager) has a component called ‘Session Manager’ that allows users to gain console-based access to VMs registered to SSM using the AWS API as a transport tunnel.
Essentially, the session interaction data is delivered to the end device via the so-called SSM messages. The advantage of this method, compared to standard SSH is that it does not require the opening of any TCP ports towards the internet, thus reducing the attack surface.
Clearly, something IS exposed to the Internet. And that is the AWS API, which – we trust – is highly secured and protected by layers of filtering and DDoS protection.

Following best-practises to secure the access to the AWS API is particularly important here, more than ever.

AWS VPC Endpoints

But how is the bastion reached by SSM and, viceversa, how does it reach SSM if there’s no public internet access?
This is achieved via VPC Endpoints (Interface). VPC Endpoints create ENIs (Elastic Network Interfaces) in one (or more) subnets that act as a proxy for the service which the Endpoint is being created for.

In this particular instance we are creating VPC endpoints for:

  • SSM Messages: to enable the flow of SSM messages (carrying the user session-interaction data)
  • EC2 Messages: same reason as above
  • SSM: to allow the SSM-agent to register on SSM

More details on the above requirement here: https://docs.aws.amazon.com/systems-manager/latest/userguide/setup-create-vpc.html

resource "aws_vpc_endpoint" "ssm_messages" {

  vpc_id            = "${aws_vpc.main.id}"

  service_name      = "com.amazonaws.eu-west-1.ssmmessages"

  vpc_endpoint_type = "Interface"

  subnet_ids = aws_subnet.private_subnets.*.id

  security_group_ids = [ aws_security_group.endpoints_sg.id ]

  private_dns_enabled = true

}

resource "aws_vpc_endpoint" "ec2messages" {

  vpc_id            = "${aws_vpc.main.id}"

  service_name      = "com.amazonaws.eu-west-1.ec2messages"

  vpc_endpoint_type = "Interface"

  subnet_ids = aws_subnet.private_subnets.*.id

  security_group_ids = [ aws_security_group.endpoints_sg.id ]

  private_dns_enabled = true

}

resource "aws_vpc_endpoint" "ssm" {

  vpc_id            = "${aws_vpc.main.id}"

  service_name      = "com.amazonaws.eu-west-1.ssm"

  vpc_endpoint_type = "Interface"

  subnet_ids = aws_subnet.private_subnets.*.id

  security_group_ids = [ aws_security_group.endpoints_sg.id ]

  private_dns_enabled = true

}

AWS EC2 Instance Profile

An EC2 instance profile is necessary to allow the EC2 instance to make API calls to the AWS SSM without the need of having to store credentials on the instance itself.

resource "aws_iam_role" "ec2_ssm_role" {
  name = "ec2-bastion-ssm-role"
  assume_role_policy = data.aws_iam_policy_document.instance_assume_role_policy.json
}

data "aws_iam_policy_document" "instance_assume_role_policy" {
  statement {
    actions = ["sts:AssumeRole"]

    principals {
      type        = "Service"
      identifiers = ["ec2.amazonaws.com"]
    }
  }
}


resource "aws_iam_instance_profile" "ec2_ssm_instance_profile" {
  name = "ec2-ssm-profile"
  role = aws_iam_role.ec2_ssm_role.name
}

resource "aws_iam_role_policy_attachment" "ec2_ssm_role_policy_attachment" {
  role       = aws_iam_role.ec2_ssm_role.name
  policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
}

Security groups

The security groups are designed so that hosts can accept TCP 22 just from the bastion, to ensure maximum isolation and reduce attack surface.

https://github.com/dustnic/terraform-aws-ssm-bastion/blob/master/secgroups.tf

Custom AMI

The bastion requires a few packages to be present to work properly and tunnel ssh connections.
https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-getting-started-enable-ssh-connections.html
The first being the amazon-ssm-agent version greater than 2.3.672.0 and the nc package that we’ll use as a socket broker.

I created a custom AMI with both requirements satisfied, as the bastion won’t have internet access to be able to install those later.

Deployment

The deployment of all the infrastructure is managed by Terraform.
You can have a look at the codebase here.
https://github.com/dustnic/terraform-aws-ssm-bastion
The Terraform plan produces a host_key.pem file that contains the private ssh key used to ssh into the hosts, and a few outputs indicating the bastion instance ID and the host(s) private_ip addresses.

bastion_instance_id = i-01bbebed1121214e9
host_private_ip = 10.0.128.218

Before we test this out from our local machine, we need to install the ssm-plugin: https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html

curl "https://s3.amazonaws.com/session-manager-downloads/plugin/latest/mac/sessionmanager-bundle.zip" -o "sessionmanager-bundle.zip"

unzip sessionmanager-bundle.zip

sudo ./sessionmanager-bundle/install -i /usr/local/sessionmanagerplugin -b /usr/local/bin/session-manager-plugin

There is one last thing we need to do if we want to make sure we can open an SSM session to our bastion instance. That is that we need to have our AWS CLI configuration in place. I conveniently use named profiles for that.

SSH over an SSM session

Now we need to configure our SSH client to use the SSM plugin as a transport mechanism via the ProxyCommand directive in the ~/.ssh/config file.

Host i-* mi-*
    ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'"

Tunneling SSH to hosts through the bastion

Now it’s time to see if we can ssh into hosts via the bastion.
To do that, we need some extra config in the ~/.ssh/config file.

Host 10.0.*
   ProxyCommand ssh ec2-user@i-01bbebed1121214e9 nc %h %p

Conclusion

In this post I have shown how console-based remote access can be achieved without exposing port 22 externally using a single bastion and multiple target hosts.
Two of the biggest advantages to SSM are that sessions activity can be logged in CloudWatch Logs/S3 and that access can be disciplined using IAM.

The approach described in this post renders those impossible as the tunneled sessions to the end-hosts are not captured, as they’re not terminal activity.

This post’s objective was more to show that it CAN be done, not that it SHOULD be done regardless of specific requirements.

One thought on “Stop exposing port 22 to the world. It’s time to rework your remote access methods

Add yours

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Blog at WordPress.com.

Up ↑

%d bloggers like this: