Quick templating with python in a shell heredoc

Say you access various data sources (primarily enviroment vars) that is to be mixed into a result output textfile. What is the quickest and most powerful way?

Replacing variable refs inside a pure text file has many solutions out there. Most ones come as compiled, extra install or limited in scope. (m4, envsubst, gomplate)

Here I will examine solving this using the python3 interpreter directly from a single shell command. No pip dependencies, no script file.

In most cases templating ends up being executed in a CI/CD docker container where you would like to avoid complex dependencies. Python3 might not be an "extra install" if you work with deployments to AWS and use either aws-cli or botocore directly.

💬 Why a heredoc and not a .py scriptfile? In my case a limited "reuse" system (gitlab include) require passing the shell commands inside yaml and no extra files allowed, and shell heredoc is quite elegant and makes you focus on compact code.

Solution with shell heredoc

The solution should be *nix, simple and sweet :

$ python3 /dev/fd/3 3<<EOF < input.txt > output.txt 
import sys
# use batteries
[sys.stdout.write(line) for line in sys.stdin]
EOF

To execute the shell heredoc we use filedescriptor /dev/fd/3. Avoid "-" since the interpreter then picks stdin (input.txt) and not /fd/0 (default <<heredoc) like some other unix tools would choose to do.

Only batteries included

You could use pip to install a package like jinja, but first lets see how only using the batteries included with python3 can work.

💡Instead using the recent python3.7+ formatting we can write an input text file like this:

# input.yaml.templ
---
apiVersion: v1
kind: Secret
metadata:
  name: secret-{env[SECRET_SUFFIX]}
data:
  username: {env[USERNAME]}
  password: {env[PASSWORD]}

Then execute :

SECRET_SUFFIX=mycredentials \
PASSWORD=password1 \
USERNAME=kjetil \
python3 /dev/fd/3 3<<EOF < input.yaml.templ 
import sys, os
vals = dict(env = os.environ)
[sys.stdout.write(line.format(**vals)) for line in sys.stdin]
EOF

I choose to "namespace" every environ key in env[key] because then I can add more datasources and avoid name conflicts.

AWS SSM Parameters in the mix

If you have botocore (pip) installed then imagine an input text file like this:

# input2.yaml.templ
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: meta_on_{env[USER]}
data:
  email: {env[USER]}@{project[subprojects/cmp/domain]}
  address: {employee[home/address]}
  city: {employee[home/city]}
  hobbies: {employee[personal/hobbies]}
  coworkers: {project[subprojects/cmp/taskforce]}

To make this sample work you must have a tree of 5 parameters in aws ssm looking like this:

/
├── company
│  └── abc
│     └── employees
│        └── kjetil
│           ├── home
│           │  ├── address
│           │  └── city
│           └── personal
│              └── hobbies
└── development
   └── project_x
      └── subprojects
         └── cmp
            ├── domain
            └── taskforce

( github/gist script for putting these into AWS )

Assume correct aws profile and execute :

AWS_PROFILE=default \
USER=kjetil \
SSM_PROJ_PATH=/development/project_x/ \
python3 /dev/fd/3 3<<EOF < input2.yaml.templ 
import sys, os, botocore.session

ssm = botocore.session.get_session().create_client('ssm')

def ssm_ps(path):
  ps = ssm.get_parameters_by_path( Path=path, Recursive=True, WithDecryption=True)
  return { p['Name'].replace(path,'') : p['Value'] 
           for p in ps['Parameters'] }

env = os.environ
vals = dict(env = env,
            employee = ssm_ps(f"/company/abc/employees/{env['USER']}/"),
            project = ssm_ps(env["SSM_PROJ_PATH"]))

[sys.stdout.write(line.format(**vals)) for line in sys.stdin]
EOF

Here vals consist of 3 "namespaces" : env, employee & project. 

# RESULT OUTPUT stdout (if you used my gist to put-parameters )
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: meta_on_kjetil
data:
  email: kjetil@sample.com
  address: Queens st.
  city: Oslo
  hobbies: code,code then code
  coworkers: per,pål,espen

Error handling

This code will NOT handle missing keys gracefully and you would just see the stacktrace ending in something like: 

• KeyError: 'env[ENV_NOT_SET]'

Next steps

Soon you would want to loop list, and options could be 

• Jinja or mako etc (pip)
• f-string eval is also a scary last resort powerful option