This post is aimed at showing how remote console-based access to cloud-resources can be achieved without exposing port 22 to the world, and – in fact – without any kind of internet access provided to your virtual resources.
This post is based on AWS technologies, which I’m most familiar with, but similar features are being rolled out by the other Cloud Providers.
The idea is to deploy a ‘bastion’ host in a private subnet on AWS and use that as an SSH proxy to access any other hosts in the same VPC and in any other private subnet, without any requirement to expose any port in any of your Security Groups or NACLs to the Internet.
Clearly, this is not meant to be a production-ready deployment, but just serve as inspiration on how things can be done differently.
The infrastructure
We have:
- a private subnet where the Bastion host is deployed
- another private subnet(s) where target hosts are deployed
- a set of VPC Endpoints (Interface-type) to allow the AWS System Manager to reach and be reached by the Bastion SG ssm-agent on port 443
- three separate security groups regulating inbound and outbound access to/from the various groups of hosts/interfaces in the infrastructure.
How it all works
AWS System Manager
AWS SSM (System Manager) has a component called ‘Session Manager’ that allows users to gain console-based access to VMs registered to SSM using the AWS API as a transport tunnel.
Essentially, the session interaction data is delivered to the end device via the so-called SSM messages. The advantage of this method, compared to standard SSH is that it does not require the opening of any TCP ports towards the internet, thus reducing the attack surface.
Clearly, something IS exposed to the Internet. And that is the AWS API, which – we trust – is highly secured and protected by layers of filtering and DDoS protection.
Following best-practises to secure the access to the AWS API is particularly important here, more than ever.
AWS VPC Endpoints
But how is the bastion reached by SSM and, viceversa, how does it reach SSM if there’s no public internet access?
This is achieved via VPC Endpoints (Interface). VPC Endpoints create ENIs (Elastic Network Interfaces) in one (or more) subnets that act as a proxy for the service which the Endpoint is being created for.
In this particular instance we are creating VPC endpoints for:
- SSM Messages: to enable the flow of SSM messages (carrying the user session-interaction data)
- EC2 Messages: same reason as above
- SSM: to allow the SSM-agent to register on SSM
More details on the above requirement here: https://docs.aws.amazon.com/systems-manager/latest/userguide/setup-create-vpc.html
resource "aws_vpc_endpoint" "ssm_messages" {
vpc_id = "${aws_vpc.main.id}"
service_name = "com.amazonaws.eu-west-1.ssmmessages"
vpc_endpoint_type = "Interface"
subnet_ids = aws_subnet.private_subnets.*.id
security_group_ids = [ aws_security_group.endpoints_sg.id ]
private_dns_enabled = true
}
resource "aws_vpc_endpoint" "ec2messages" {
vpc_id = "${aws_vpc.main.id}"
service_name = "com.amazonaws.eu-west-1.ec2messages"
vpc_endpoint_type = "Interface"
subnet_ids = aws_subnet.private_subnets.*.id
security_group_ids = [ aws_security_group.endpoints_sg.id ]
private_dns_enabled = true
}
resource "aws_vpc_endpoint" "ssm" {
vpc_id = "${aws_vpc.main.id}"
service_name = "com.amazonaws.eu-west-1.ssm"
vpc_endpoint_type = "Interface"
subnet_ids = aws_subnet.private_subnets.*.id
security_group_ids = [ aws_security_group.endpoints_sg.id ]
private_dns_enabled = true
}
AWS EC2 Instance Profile
An EC2 instance profile is necessary to allow the EC2 instance to make API calls to the AWS SSM without the need of having to store credentials on the instance itself.
resource "aws_iam_role" "ec2_ssm_role" {
name = "ec2-bastion-ssm-role"
assume_role_policy = data.aws_iam_policy_document.instance_assume_role_policy.json
}
data "aws_iam_policy_document" "instance_assume_role_policy" {
statement {
actions = ["sts:AssumeRole"]
principals {
type = "Service"
identifiers = ["ec2.amazonaws.com"]
}
}
}
resource "aws_iam_instance_profile" "ec2_ssm_instance_profile" {
name = "ec2-ssm-profile"
role = aws_iam_role.ec2_ssm_role.name
}
resource "aws_iam_role_policy_attachment" "ec2_ssm_role_policy_attachment" {
role = aws_iam_role.ec2_ssm_role.name
policy_arn = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
}
Security groups
The security groups are designed so that hosts can accept TCP 22 just from the bastion, to ensure maximum isolation and reduce attack surface.
https://github.com/dustnic/terraform-aws-ssm-bastion/blob/master/secgroups.tf
Custom AMI
The bastion requires a few packages to be present to work properly and tunnel ssh connections.
https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-getting-started-enable-ssh-connections.html
The first being the amazon-ssm-agent version greater than 2.3.672.0 and the nc package that we’ll use as a socket broker.
I created a custom AMI with both requirements satisfied, as the bastion won’t have internet access to be able to install those later.
Deployment
The deployment of all the infrastructure is managed by Terraform.
You can have a look at the codebase here.
https://github.com/dustnic/terraform-aws-ssm-bastion
The Terraform plan produces a host_key.pem file that contains the private ssh key used to ssh into the hosts, and a few outputs indicating the bastion instance ID and the host(s) private_ip addresses.
bastion_instance_id = i-01bbebed1121214e9 host_private_ip = 10.0.128.218
Before we test this out from our local machine, we need to install the ssm-plugin: https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html
curl "https://s3.amazonaws.com/session-manager-downloads/plugin/latest/mac/sessionmanager-bundle.zip" -o "sessionmanager-bundle.zip" unzip sessionmanager-bundle.zip sudo ./sessionmanager-bundle/install -i /usr/local/sessionmanagerplugin -b /usr/local/bin/session-manager-plugin
There is one last thing we need to do if we want to make sure we can open an SSM session to our bastion instance. That is that we need to have our AWS CLI configuration in place. I conveniently use named profiles for that.
SSH over an SSM session
Now we need to configure our SSH client to use the SSM plugin as a transport mechanism via the ProxyCommand directive in the ~/.ssh/config file.
Host i-* mi-*
ProxyCommand sh -c "aws ssm start-session --target %h --document-name AWS-StartSSHSession --parameters 'portNumber=%p'"
Tunneling SSH to hosts through the bastion
Now it’s time to see if we can ssh into hosts via the bastion.
To do that, we need some extra config in the ~/.ssh/config file.
Host 10.0.* ProxyCommand ssh ec2-user@i-01bbebed1121214e9 nc %h %p
Conclusion
In this post I have shown how console-based remote access can be achieved without exposing port 22 externally using a single bastion and multiple target hosts.
Two of the biggest advantages to SSM are that sessions activity can be logged in CloudWatch Logs/S3 and that access can be disciplined using IAM.
The approach described in this post renders those impossible as the tunneled sessions to the end-hosts are not captured, as they’re not terminal activity.
This post’s objective was more to show that it CAN be done, not that it SHOULD be done regardless of specific requirements.
