Migrating SQL Server container images to the Github Container Registry

A couple of months ago Docker announced that they would be implementing a 6 month retention policy for unused images in the Docker Hub.

This was due to kick in on the 1st of November but has now been pushed back until mid 2021.

I’ve had multiple Windows SQL Server container images up on the Docker Hub for years now. It’s been a great platform and I’m very thankful to them for hosting my images.

That being said, I want to make sure that the images that I’ve built are always going to be available for the community so I have pushed my SQL Server images to the Github Container Registry.

In the Docker Hub I have the following public SQL Server images: –

dbafromthecold/sqlserver2012express:rtm
dbafromthecold/sqlserver2012dev:sp4
dbafromthecold/sqlserver2014express:rtm
dbafromthecold/sqlserver2014dev:sp2
dbafromthecold/sqlserver2016dev:sp2

These images won’t be removed (by myself) and I will update this post in the event of them being removed.

But they are now available on the Github Container Registry: –

ghcr.io/dbafromthecold/sqlserver2012:express
ghcr.io/dbafromthecold/sqlserver2012:dev
ghcr.io/dbafromthecold/sqlserver2014:express
ghcr.io/dbafromthecold/sqlserver2014:dev
ghcr.io/dbafromthecold/sqlserver2016:dev

Bit of a disclaimer with these images…they’re LARGE! The 2012 dev image is ~20GB which is gigantic for a container image. So if you going to use them (for dev and test only please 🙂 ) you’ll need to pre-pull them before running any containers.

Thanks for reading!

The SQL Server and Containers Guide

I’ve been blogging about running SQL Server in Docker containers for a while now and, to be honest, my blogs are scattered over a few years and some need to be archived as they’re out of date.

So what I wanted to do was have one place where I could collate all the blogs I’ve written about running SQL Server in a container. This would make it easy for people to access information and make it easy for me to keep it all up-to-date as well.

So introducing, The SQL Server and Containers Guide!

This wiki contains everything needed to get up and running with SQL Server in Docker containers, and it delves a little bit deeper as well.

Topics covered are: –

  • Quick Start – Get up and running with SQL in a container
  • Running SQL in container – Explore the ins and outs of running SQL in a container
  • Persisting data – Have a look at the options for persisting data in a container
  • Docker images – Build your own custom SQL images
  • Container networking – Explore the default docker network and create a custom network
  • Docker Compose – Use Docker Compose to spin SQL up in a container
  • Docker commands – Details of various Docker commands

Each post has code and screenshots to walk through each topic…and the Github repository contains all the code from the posts so that it can be pulled down locally.

I’ll be adding to this some more over the next few weeks…Rob Sewell (b|t) awesomely contributed code to generate Azure Data Studio Notebooks from the wiki which I may (ok, definitely) have broken. But once I’ve (or more likely Rob) has fixed that…they’ll be added into the repository.

If there’s anything missing please feel free to contact me via dbafromthecold@gmail.com or submit a PR!

Thanks for reading!

SQL Server and Docker Compose

I used to think that Docker Compose was used solely to spin up multiple containers, in fact I blogged about doing just that here.

That opinion changed when I went to DockerCon in 2018 and had a chance to speak to some Docker Captains who told me that they used compose for everything!

And it makes sense, let’s have a look at spinning up one container running SQL Server 2019: –

docker run -d -p 15789:1433 `
--env ACCEPT_EULA=Y `
--env MSSQL_SA_PASSWORD=Testing1122 `
--name testcontainer `
mcr.microsoft.com/mssql/server:2019-CU5-ubuntu-18.04

Quite a bit to type there, no? Do we really want to be typing that out every time we run a container?

And it gets even worse if we want to persist our databases from one container to another: –

docker container run -d `
-p 15789:1433 `
--volume systemdbs:/var/opt/mssql `
--volume userdbs:/var/opt/sqlserver `
--env MSSQL_SA_PASSWORD=Testing1122 `
--env ACCEPT_EULA=Y `
--env MSSQL_BACKUP_DIR="/var/opt/sqlserver" `
--env MSSQL_DATA_DIR="/var/opt/sqlserver" `
--env MSSQL_LOG_DIR="/var/opt/sqlserver" `
--name testcontainer `
mcr.microsoft.com/mssql/server:2019-CU5-ubuntu-18.04

That’s a lot of typing! And if we try to create a database with the default values set in that statement, we’ll get the following error: –

CREATE FILE encountered operating system error 2(The system cannot find the file specified.) while attempting to open or create the physical file ‘/var/opt/sqlserver/testdatabase.mdf’.

This is because SQL in 2019 runs as non-root. This is a good thing but it means that after the container comes up, we have to run: –

docker exec -u 0 testcontainer bash -c "chown mssql /var/opt/sqlserver"

The solution here is to create a custom image with the volume created and permissions set.

But wouldn’t it be easier to just have to run one command to spin up a custom 2019 image, with volumes created and permissions set?

Enter Docker Compose.

I’ve created a GitHub repository here with all the necessary files: –
https://github.com/dbafromthecold/SqlServerDockerCompose

If we clone that repo down, we’ll get the following: –

Let’s go through each of the files

.gitignore
Standard ignore file, this is to prevent the sapassword.env file from being uploaded to Github

docker-compose.yaml
Compose file that when executed will reference our dockerfile and build us a custom image

dockerfile
File to create a custom SQL 2019 image

sapassword.env
Environment variable file to contain our SA password. We’ll need to create this file, it’s not in the repo

sqlserver.env
Environment variable file that contains all the environment variables required to spin up SQL Server in a container

Let’s dive in a little deeper and first have a look at the dockerfile: –

# build from the Ubuntu 18.04 image
FROM ubuntu:18.04

# create the mssql user
RUN useradd -u 10001 mssql

# installing SQL Server
RUN apt-get update && apt-get install -y wget software-properties-common apt-transport-https
RUN wget -qO- https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
RUN add-apt-repository "$(wget -qO- https://packages.microsoft.com/config/ubuntu/18.04/mssql-server-2019.list)"
RUN apt-get update && apt-get install -y mssql-server

# creating directories
RUN mkdir /var/opt/sqlserver
RUN mkdir /var/opt/sqlserver/data
RUN mkdir /var/opt/sqlserver/log
RUN mkdir /var/opt/sqlserver/backup

# set permissions on directories
RUN chown -R mssql:mssql /var/opt/sqlserver
RUN chown -R mssql:mssql /var/opt/mssql

# switching to the mssql user
USER mssql

# starting SQL Server
CMD /opt/mssql/bin/sqlservr

This file when executed is going to create a custom SQL 2019 image, not from the microsoft images but installed via apt-get (the way you would install SQL on Linux).

It’s based on the Ubuntu 18.04 image and the steps are: –

  1. Pull down the Ubuntu 18.04 image and base this new image off it
  2. Create the mssql user
  3. Install SQL Server as you would on Linux, detailed instructions here
  4. Create the required directories
  5. Change the owner of those directories to the mssql user
  6. Switch over to run the next command as the mssql user
  7. Start SQL Server

Ok, cool. Let’s now have a look at the docker-compose.yaml file: –

version: '3.7'
services:
    sqlserver1:
        build: 
          context: .
          dockerfile: dockerfile
        ports:  
          - "15789:1433"
        env_file:
          - sqlserver.env
          - sapassword.env
        volumes: 
          - sqlsystem:/var/opt/mssql/
          - sqldata:/var/opt/sqlserver/data
          - sqllog:/var/opt/sqlserver/log
          - sqlbackup:/var/opt/sqlserver/backup
volumes:
  sqlsystem:
  sqldata:
  sqllog:
  sqlbackup:

Stepping through this we: –

  1. Define a service called sqlserver1, setting a build context to the current directory and specifying our dockerfile
  2. Set our ports, mapping 15789 on the host to 1433 in the container
  3. Specify our environment variable files
  4. Then set our volumes, matching the directories created in the dockerfile

And finally, let’s have a look at the two environment variable files: –

sqlserver.env

ACCEPT_EULA=Y
MSSQL_DATA_DIR=/var/opt/sqlserver/data
MSSQL_LOG_DIR=/var/opt/sqlserver/log
MSSQL_BACKUP_DIR=/var/opt/sqlserver/backup

sapassword.env

MSSQL_SA_PASSWORD=Testing1122

The SA password is set in a separate file so that we don’t end up putting it somewhere public 🙂
The other file can contain any environment variable for SQL Server, a full list is here.

Awesome stuff. OK, now we can run: –

docker-compose up -d

And we can check the objects created by compose by running: –

docker network ls
docker volume ls
docker image ls
docker container ls

There we can see our custom network, volumes, image, and container up and running!

So we’re good to do our work on SQL Server 2019 and when we’re finished we can just run: –

docker-compose down

That’ll delete our custom network and the container but we’ll still have our custom image and volumes, ready for next time we want to do some work against SQL Server 2019.

Thanks for reading!

Using volumes in SQL Server 2019 non-root containers

I’ve seen a few people online asking how to use docker named volumes with the new SQL Server 2019 RTM images. Microsoft changed the way SQL runs within a container for the RTM versions, SQL no longer runs as the root user.

This is a good thing but does throw up some issues when mounting volumes to create databases on.

Let’s run through what the issue is and how to overcome it.

Run a container from the 2019 GDR1 ubuntu image: –

docker container run -d `
-p 15789:1433 `
--volume sqlserver:/var/opt/sqlserver `
--env MSSQL_SA_PASSWORD=Testing1122 `
--env ACCEPT_EULA=Y `
--env MSSQL_BACKUP_DIR="/var/opt/sqlserver" `
--env MSSQL_DATA_DIR="/var/opt/sqlserver" `
--env MSSQL_LOG_DIR="/var/opt/sqlserver" `
--name testcontainer `
mcr.microsoft.com/mssql/server:2019-GDR1-ubuntu-16.04

What we’re going here is mounting a named volume called sqlserver to /var/opt/sqlserver within the container. Then we’re setting the default data, log, and backup location to /var/opt/sqlserver/.

Now if we try and create a database using those defaults: –

CREATE DATABASE [TestDatabase];
GO

Msg 5123, Level 16, State 1, Line 1
CREATE FILE encountered operating system error 2(The system cannot find the file specified.) while attempting to open or create the physical file ‘/var/opt/sqlserver/testdatabase.mdf’.
Msg 1802, Level 16, State 4, Line 1
CREATE DATABASE failed. Some file names listed could not be created. Check related errors.

We get an error message as the SQL instance within the container does not have access to that location because it’s running as the mssql user.

We need to grant the mssql user access to that folder: –

docker exec -u 0 testcontainer bash -c "chown mssql /var/opt/sqlserver"

This will make the mssql user the owner of that folder. -u 0 sets the command to run as the root user and it has access to be able to change the owner of the folder. For more info on docker exec click here.

So we can now create the database: –

However, we would have to run that command every time we spin up a container with named volumes mounted. A better way would be to create a custom image from a Dockerfile that has created that folder within the container and granted the mssql user access: –

FROM mcr.microsoft.com/mssql/server:2019-GDR1-ubuntu-16.04

USER root

RUN mkdir /var/opt/sqlserver

RUN chown mssql /var/opt/sqlserver

ENV MSSQL_BACKUP_DIR="/var/opt/sqlserver"
ENV MSSQL_DATA_DIR="/var/opt/sqlserver"
ENV MSSQL_LOG_DIR="/var/opt/sqlserver"

USER mssql

CMD /opt/mssql/bin/sqlservr

We’re using the USER command to switch to the root user in order to grant access to the folder and then switching back to the mssql user to run SQL.

Create the custom image: –

docker build -t custom2019image .

Now we can run a container from that image: –

docker container run -d `
-p 15789:1433 `
--volume sqlserver:/var/opt/sqlserver `
--env MSSQL_SA_PASSWORD=Testing1122 `
--env ACCEPT_EULA=Y `
--name testcontainer `
custom2019image

And create the database without having to run anything else: –

CREATE DATABASE [TestDatabase];
GO

Hope that helps!

Using the GitHub Package Registry to store container images


UPDATE – October 2020 – Github has now released the Github Container Registry which can be used to store container images. For more information, see here


The GitHub Package Registry is available for beta testing and allows us to store container images in it, basically giving us the same functionality as the Docker Hub.

However the Docker Hub only allows for one private repository per free account whereas the Github package registry is completely private! Let’s run through a simple demo to create a registry and upload an image.

First thing to do is create a personal access token in GitHub. Go to Settings > Developer Settings > Personal Access Tokens

Ensure that the token has the rights set above and click Generate Token

Now we can use that token to login to the package registry: –

TOKEN=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
docker login docker.pkg.github.com -u dbafromthecold -p $TOKEN

Search for a test image. I’m going to use the busybox image which is 2MB: –

docker search busybox

Then pull the image down: –

docker pull busybox:latest

Tag the image with the repo name to be push to. The format is docker.pkg.github.com/USERNAME/REPOSITORY/IMAGE:TAG

docker tag busybox:latest docker.pkg.github.com/dbafromthecold/testrepository/busybox:latest

N.B. – the repo used has to already exist within your github account

Now push the image to the GitHub Package repository: –

docker push docker.pkg.github.com/dbafromthecold/testrepository/busybox:latest

And then you should be able to see the package in GitHub: –

Thanks for reading!