Day 4

Production & Automation

So you are running a server

What could go wrong?

⚠️Failures⛔️

• Your service can crash
• Data can be lost
  • Hardware failure
  • User/admin mistake
• A security vulnerability can be discovered
• The SSL certificate can expire

Manual Maintenance

• Service crash: Restart the service
• Data loss: Restore data from backup
• Security vulnerability: Upgrade the service

Manual Maintenance

• ➡️ Service crash: Restart the service
• Data loss: Restore data from backup
• Security vulnerability: Upgrade the service

Exercise

• SSH into your server
• Run docker container restart <CONTAINER_NAME> to restart a container
• Run docker compose restart to restart all compose containers
• Run sudo shutdown -r now to restart the whole server

Manual Maintenance

• Service crash: Restart the service
• ➡️ Data loss: Restore data from backup
• Security vulnerability: Upgrade the service

Exercise - restoring from backup

1. Deploy todo app, make some todos
2. Make a backup
3. Delet this
4. Relaunch the app
5. Restore from backup
6. Pray

Exercise - Deploy todo list

• docker run -d --name my_todoapp -p 8000:8000 -v todo_db:/data prologic/todo
• Mess around with it at http://<<YOUR_SERVER>>:8000

Exercise - Back it up

SSH into the server, then:

sudo apt install rsync
# Identify Docker volumes 
ls /var/lib/docker/volumes
# Then, disconnect from your server
exit

Then, to copy the volume data to your local computer: (On a Mac? run brew install rsync)

rsync -aP root@<YOUR_DOMAIN>:/var/lib/docker/volumes/todo_db .

Or, on your server:

sudo cp -r /var/lib/docker/volumes/todo_db ~/todo_db

rsync syntax

• rsync <flags> <source> <destination>
• source/destination: <server>:<path> or <local path>

Exercise - Delete everything

On your server:

# Stop the app
docker container stop my_todoapp
# Delete the app container
docker container rm my_todoapp
docker container ls
docker volume ls
# Delete the app volume
docker volume rm todo_db
docker volume ls

Exercise - Re-deploy

• docker run -d --name my_todoapp -p 8000:8000 -v todo_db:/data prologic/todo
• Check that it’s available at http://<SERVER>:8000

Exercise - Restore

Choose an option:

# Option A: Copy data from your local environment
rsync -aP todo_db root@<YOUR_SERVER>:/var/lib/docker/volumes/
# Option B: restore data from the copy on the server
sudo cp -r ~/todo_db /var/lib/docker/volumes/

Then:

sudo chown -R $USER:$USER /var/lib/docker/volumes/todo_db
# Restart the service
docker container restart my_todoapp

Then, check that it’s restored at http://<SERVER>:8000

Manual Maintenance

• Service crash: Restart the service
• Data loss: Restore data from backup
• ➡️ Security vulnerability: Upgrade the service

Exercise - Security Update

• Run a service: docker run --name nginx -v content:/usr/share/nginx/html -p 8080:80 -d nginx:1.26
• Check the version: http://<YOUR_SERVER>:8080/error
• Copy some stuff to the service: docker cp /etc/motd nginx:/usr/share/nginx/html/motd.html
• Confirm that it’s available: http://<YOUR_SERVER>:8080/motd.html

Exercise - Security Update

# Stop the service
docker container stop nginx
# Remove the old version
docker container rm nginx
# Launch an updated version of the service
docker run --name nginx -v content:/usr/share/nginx/html \
  -p 8080:80 -d nginx:1.29

Then, confirm that the content is available at http://<YOUR_SERVER>:8080/motd.html, and check the version at http://<YOUR_SERVER>:8080/error

BREAK

Autonomous infrastructure isn’t autonomous

Servers are just machines

• Machines require maintenance to keep running
• Machines break and require repair
• This is specialized labor, it takes time and attention

Are we technocrats?

Traditional Maintenance

Traditional Maintenance

Traditional Maintenance

• Advantages
  • Very flexible
  • Responsive, tight loop between making a change and seeing result
• Disadvantages
  • Requires lots of skill and experience
  • Each environment is unique, requires individual consideration and attention
  • Doesn’t scale well

Guided Maintenance

Guided Maintenance

• Advantages
  • Accessible, much less skill/experience needed
  • More consistent outcomes
• Disadvantages
  • Configuration drift - small deviations from the script compound over time
  • Still requires specific human attention, limits scale

Infrastructure as Code

Infrastructure as Code

services:
  server:
    image: docker.gitea.com/gitea:nightly
    container_name: gitea
    environment:
      - GITEA__database__DB_TYPE=postgres
      - GITEA__database__HOST=db:5432
    restart: always
    networks:
      - gitea
    volumes:

Infrastructure as Code

• Advantages
  • Low skill/experience required
  • Consistent outcome
  • Repeatable
  • Very scalable
  • Fungible servers - “cattle vs pets”
  • Facilitates collaboration

Infrastructure as Code

• Disadvantages
  • Not flexible
  • Intolerant of failure
  • Slower to do anything the first time

Automation?

Automation?

• Automation can make people disposable, for better or worse
• Labor can never be fully eliminated
• We can reduce unnecessary repetition
• Code turns technical labor into a post-scarcity commodity
  • Write once, copy infinitely
  • Expands who can use a system
  • Flattens technocratic hierarchies

Limitations

• Each Docker app has to be configured differently, no standardization
• Each Docker app has to be backed up differently
• Each app you host has to be integrated with your ingress controller
• Checking for new upgrades and applying them is toil

Co-op Cloud

• https://coopcloud.tech

Exercise

• Prepare SSH for Co-Op Cloud connections
• nano ~/.ssh/config

Host <YOUR_SERVER>.autonomous.rodeo
Hostname <YOUR_SERVER>.autonomous.rodeo
User root
IdentityFile ~/.ssh/<YOUR_SSH_PRIVATE_KEY>

• Test: ssh <YOUR_SERVER>.autonomous.rodeo

Exercise - Deploy Nextcloud

• On your local machine: ssh-keygen -p
  • ./.ssh/<YOUR_PRIVATE_KEY_NAME>
• https://docs.coopcloud.tech/operators/tutorial/

Exercise - Coop Cloud Backups

• https://git.coopcloud.tech/coop-cloud/backup-bot-two#deployment

Exercise

• abra app new backup-bot-two
• abra app config <APP_NAME>
• abra app secret generate -a <BACKUPBOT_NAME>
• abra app deploy <APP_NAME>

Exercise

• Look around
• abra app ls
• abra app ls -S

Exercise

• Trigger a backup manually
• abra app run <APP_NAME> app backup --host <YOUR_SERVER>.autonomous.rodeo create

Exercise

• See what backups exist
• abra app run <APP_NAME> app backup --host <YOUR_SERVER>.autonomous.rodeo snapshots

Exercise

Restore a snapshot

abra app run <APP_NAME> app backup \
  --host <YOUR_SERVER>.autonomous.rodeo \
  restore \
  --snapshot fea04a7c064452b730445d81027ba185aa6626536b2c929d7a7c9fdc998b28c6

Extra Credit 🏅

• Find another service at https://recipes.coopcloud.tech/ and deploy it.