Flask and Redis
Up to this point, we have been interacting with a shared Redis database instance running on the class ISP server. Here, we will each work with our own containerized instance of Redis, figure out how to interact with it by forwarding the port, update the command to mount a host directory to persist data, and connect it to our Flask app. After going through this module, students should be able to:
Start a Redis container, connecting the appropriate inside port to a port on isp02.
Mount a host volume on isp02 inside the Redis container to save data between restarts of Redis itself.
Connect to the container from within a Flask application.
Start a Redis Container
Docker Hub has a wealth of official, public images. It is a good idea to pull the existing Redis image rather than build it ourself, because it has all of the functionality we need as a base image.
Pull the official Redis image for version 6:
[isp02]$ docker pull redis:6
6: Pulling from library/redis
ae13dd578326: Pull complete
e6f25d21ebb3: Pull complete
601cc6106ba1: Pull complete
5b8be2fd806e: Pull complete
950c3791111a: Pull complete
567b7ad78092: Pull complete
Digest: sha256:81b50efa808d72f7965d4deecea18e42cab2fec25fafca447eb4bda615b9c8e4
Status: Downloaded newer image for redis:6
docker.io/library/redis:6
Start the Redis server in a container:
[isp02]$ docker run -p <your-redis-port>:6379 --name=<your_name>-redis redis:6
1:C 31 Mar 2021 16:48:11.939 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
...
1:M 31 Mar 2021 16:48:11.972 * Ready to accept connections
The above command will start the Redis container in the foreground which can be
helpful for debugging, seeing logs, etc. However, you will have the need to
start it in the background (detached or daemon mode). You can do that by adding the
-d
flag:
[isp02]$ docker run -d -p <your-redis-port>:6379 --name=<your_name>-redis redis:6
3a28cb265d5e09747c64a87f904f8184bd8105270b8a765e1e82f0fe0db82a9e
At this point, Redis is running and available from the isp02 host, but let’s explore the persistence of the data in Redis.
We can open a Python shell, connect to our Redis container, and add some data:
>>> import redis
>>> rd = redis.Redis(host='127.0.0.1', port=<your_redis_port>, db=0)
# create some data
>>> rd.set('k', 'v')
True
# check that we can read the data
>>> rd.get('k')
b'v'
If we exit our Python shell and then go into a new Python shell and connect to Redis, what do we see?
>>> import redis
>>> rd = redis.Redis(host='127.0.0.1', port=<your_redis_port>, db=0)
# previous data is still there
>>> rd.get('k')
b'v'
Great! Redis has persisted our data across Python sessions. That was one of our goals. But what happens if we shut down the Redis container itself?
Let’s find out by killing our Redis container and starting a new one.
# shut down the existing redis container using the name you gave it
[isp02]$ docker rm -f <your_name>-redis
# start a new redis container
[isp02]$ docker run -d -p <your-redis-port>:6379 --name=<your_name>-redis redis:6
Now go back into the Python shell and connect to Redis:
>>> import redis
>>> rd = redis.Redis(host='127.0.0.1', port=<your_redis_port>, db=0)
# previous data is gone!
>>> rd.get('k')
# no keys at all!
>>> rd.keys()
[]
Oops! All the data that was in Redis is gone. The problem is we are not permanently persisting the Redis data across different Redis containers. But wasn’t that the whole point of using a database? Are we just back to where we started?
Actually, we only need two small changes to the way we are running the Redis container to make the Redis data persist across container executions.
Container Bind Mounts
A container bind mount (or just “mount” for short) is a way of replacing a file or directory in a container image with a file or directory on the host file system in a running container.
Bind mounts are specified with the -v
flag to the docker run
statement. The full syntax
is
[isp02]$ docker run -v <host_path>:<container_path>:<mode> ...*additional docker run args*...
where:
<host_path>
and<container_path>
are absolute paths in the host (respectively, container) file system and
<mode>
can take the value ofro
for a read-only mount andrw
for a read-write mount.
Note that mode
is optional and defaults to read-write.
It is important to keep the following in mind when using bind mounts:
If the container image originally contained a file or directory at the
<container_path>
these will be replaced entirely by the contents of<host_path>
.If the container image did not contain contents at
<container_path>
the mount will still succeed and simply create a new file/directory at the path.If the
<mode>
is read-write (the default), any changes made by the running container will be reflected on the host file system. Note that the process running in the container still must have permission to write to the path.If
<host_path>
does not exist on the host, Docker will create a directory at the path and mount it into the container. This may or may not be what you want.
Persisting Data Across Redis Containers
We can use bind mounts to persist Redis data across container executions: the key point is that Redis can be started in a mode so that it periodically writes all of its data to the host.
From the Redis documentation, we see that we need to set the --save
flag when starting Redis so
that it writes its dataset to the file system periodically. The full syntax is:
--save <frequency> <number_of_backups>
where <frequency>
is an integer, in seconds. We’ll instruct Redis to write its data to the file
system every second, and we’ll keep just one backup.
Entrypoints and Commands in Docker Containers
Let’s take a moment to revisit the difference between an entrypoint and a
command in a Docker container image. When executing a container from an image, Docker uses both
an entrypoint
and an (optional) command
to start the container. It combines the two using
concatenation, with entrypoint
first, followed by command
.
When we use docker run
to create and start a container from an existing image, we can choose to
override either the command or the entrypoint that may have been specified in the image. Any string
<string>
passed after the <image>
in the statement:
[isp02]$ docker run <options> <image> <string>
will override the command
specified in the image, but the original entrypoint set for the image
will still be used.
A common pattern when building Docker images is to set the entrypoint
to the primary program,
and set the command
to a default set of options or parameters to the program.
Consider the following simple example:
# Dockerfile
FROM ubuntu
ENTRYPOINT ["ls"]
CMD ["-l"]
If I built and tagged this image as jstubbs/ls
, then
# run with the default command, equivalent to "ls -l"
[isp02]$ docker run --rm -it jstubbs/ls
total 48
lrwxrwxrwx 1 root root 7 Jan 5 16:47 bin -> usr/bin
drwxr-xr-x 2 root root 4096 Apr 15 2020 boot
drwxr-xr-x 5 root root 360 Mar 23 18:37 dev
drwxr-xr-x 1 root root 4096 Mar 23 18:37 etc
drwxr-xr-x 2 root root 4096 Apr 15 2020 home
. . .
# override the command, but keep the entrypoint; equivalent to running "ls -a" (note the lack of "-l")
[isp02]$ docker run --rm -it jstubbs/ls -a
. .dockerenv boot etc lib lib64 media opt root sbin sys usr
.. bin dev home lib32 libx32 mnt proc run srv tmp var
# override the command, specifying a different directory
[isp02]$ docker run --rm -it jstubbs/ls /root -la
total 16
drwx------ 2 root root 4096 Jan 5 16:50 .
drwxr-xr-x 1 root root 4096 Mar 23 18:38 ..
-rw-r--r-- 1 root root 3106 Dec 5 2019 .bashrc
-rw-r--r-- 1 root root 161 Dec 5 2019 .profile
Modifying the Command in the Redis Container
The official redis container image provides an entrypoint which starts the redis server (check out the Dockerfile if you are interested.).
Since the save
option is a parameter, we can set it when running the redis server container
by simply appending it to the end of the docker run
command; that is,
[isp02]$ docker run <options> redis:6 --save <options>
Combining --save
and Mounts for a Complete Solution
With save
, we can instruct Redis to write the data to the file system, but we still need to
save the files across container executions. That’s where the bind mount comes in. But how do we know
which directory to mount into Redis? Fortunately, the Redis documentation tells us what we need to know:
Redis writes data to the /data
directory in the container.
Putting all of this together, we can update the way we run our Redis container as follows:
[isp02]$ docker run -d -p <your-redis-port>:6379 -v </path/on/host>:/data --name=<your_name>-redis redis:6 --save 1 1
Tip
You can use the ($pwd)
shortcut for the present working directory.
For example, I might use:
[isp02]$ docker run -d -p 6379:6379 -v $(pwd)/data:/data:rw --name=joe-redis redis:6 --save 1 1
Now, Redis should periodically write all of its state to the data
directory. You should see a
file called dump.rdb
in the directory because we are using the default persistence mechanism
for Redis. This will suffice for our purposes, but Redis has other options for persistence which
you can read about here if interested.
Exercise 1
Test out persistence of your Redis data across Redis container restarts by starting a new Redis container using the method above, saving some data to it in a Python shell, shutting down the Redis container and starting a new one, and verifying back in the Python shell that the original data is still there.
Using Redis in Flask
Using Redis in our Flask apps is identical to using it in the Python shells that we have been using
to explore with. We simply create a Python Redis client object using the redis.Redis()
constructor.
Since we might want to use Redis from different parts of the code, we’ll create a function for
generating the client:
1 def get_redis_client():
2 return redis.Redis(host='127.0.0.1', port=<your_port>, db=0)
Exercise 2
Note
This exercise will be part of the next home work assignment.
In the last module, we wrote a program to read the Meteorite Landings data (i.e., the
Meteorite_Landings.json
file from Unit 4) into Redis. In this exercise, let’s turn this program
into a Flask API with one route that handles both a POST and a GET.
Use
/data
as the URL path for the one route.A POST request to
/data
should load the Meteorite Landings data into Redis.A GET request to
/data
should read the data out of Redis and return it as a JSON list.
For bonus points, implement an optional start
query parameter that takes an integer and returns the
Meteorite Landing data starting at the start
index. Make sure to handle the case where start
is provided but is not an integer!