Python Environment Variables

As a modern application, yours always deals with credentials, secrets, and configurations to connect to other services like an authentication service, database, cloud services, microservices, etc.

Keeping your username, password, and other login information in the source code of your application is not a good idea because they could be exposed if you share or publish the programme.

Before sharing the code, you must delete or remark the credentials, which requires more work from you. Also, you might eventually forget to do it. The credentials you provide to the application via the command line parameters may be visible to others as well as to you.

The source code should not also contain hard codes for the setups of the services, such as API endpoints and database URLs. The issue is that you must rewrite the code each time you change or update the configurations, which could result in further mistakes.

How should we solve this issue using environment variables?

Let’s deal with it.

What are Environment Variables?
Types of Environment Variables
Accessing environment variables using Python
Python environment variables examples
Setting Environment Variables with .env file
Use cases of Environment Variables

What are Environment Variables?

The variable that is connected to an environment is referred to as an environment variable. In other words, the external variables you keep outside of your program will affect how your application behaves.

Environment variables are variables that are dynamically available to your programme or application while it is running. Values of these variables may originate from text files, outside secret managers, calling scripts, etc.

The most important fact is the value of these environment variables is not hardcoded into your application. They are genuinely dynamic and adaptable to the setting in which your software is running.

Example:

When I use my computer to work, it is my environment. It has a few variables that I may utilise on my machine globally. If you use Windows, you can search for environment variables in system properties and a list of them will then show.

Environment Variable

It contains user and system environment variables and key(variable name) value(variable value) pairs of all the environment variables. Here, the PATH is a general environment variable used by OS, specifying a set of directories where executable programs are located.

What is an .env file?

A text file called a .env file stores key-value pairs for each environment variable your application needs. You can use environment variables for local development using a .env file without contaminating the global environment namespace.

It's an easy approach to save global variables that you need but doesn't want publicly available online such as storing private data. For example:

Database login information
Username and password
API tokens
Secret codes
Crypto Wallet keys

Here is an example of a typical .env file:

VAR_UNO = SOME_KEY_HERE
VAR_DOS = SOME_OTHER_KEY_HERE

Here, VAR_UNO,VAR_DOS are keys and SOME_KEY_HERE, SOME_OTHER_KEY_HERE are values.

Types of Environment Variables

In Python, there are three types of environment variables that can be used to configure a program:

1. User environment variables

These are variables that are set by the user on their system. They are specific to the user and are not visible to other users of the system. In Python, you can access user environment variables using the os.environ dictionary.

import os

# Get the value of a user environment variable
my_var = os.environ.get('MY_VAR')

# Set the value of a user environment variable
os.environ['MY_VAR'] = 'my_value'

In this example, we use the os.environ dictionary to get and set the value of a user environment variable named "MY_VAR". If the variable is not set, os.environ.get() will return None.

2. System environment variables

These are variables that are set at the system level and are available to all users of the system. They are typically used to store configuration information that is needed by many different programs. In Python, you can access system environment variables using the os.environ dictionary.

import os

# Get the value of a system environment variable
my_var = os.environ.get('PATH')

# Set the value of a system environment variable (not recommended)
os.system('export MY_VAR=my_value')

In this example, we use the os.environ dictionary to get the value of a system environment variable named "PATH". System environment variables are set at the system level, so we can't set them directly in Python. Instead, we can use the os.system() function to execute a shell command that sets the variable. However, it is generally not recommended to set system environment variables this way.

3. Virtual environment variables

These are variables that are specific to a virtual environment created by a tool like virtualenv. Virtual environments allow you to create a self-contained environment that has its own set of dependencies and configurations. In Python, you can access virtual environment variables using the os.environ dictionary, just like user and system environment variables.

import os

# Get the value of a virtual environment variable
my_var = os.environ.get('MY_VAR')

# Set the value of a virtual environment variable
os.environ['MY_VAR'] = 'my_value'

In this example, we use the os.environ dictionary to get and set the value of a virtual environment variable named "MY_VAR". Virtual environment variables are specific to the virtual environment created by a tool like virtualenv, and are stored in the activate script for the virtual environment.

Accessing environment variables using Python

In Python, all environment variables are stored in the os.environ dictionary. The use of this conventional dictionary syntax is the simplest method for retrieving a variable from within your programme. You can use this method, for instance, to obtain the environment variable USER:

import os
user = os.environ['USER']

With this approach, Python will throw a KeyError exception if you attempt to import a nonexistent environment variable:

database_url = os.environ['DATABASE_URL']

Output:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/miguel/.pyenv/versions/3.8.6/lib/python3.8/os.py", line 675, in __getitem__
    raise KeyError(key) from None
KeyError: 'DATABASE_URL'

1. How to get an environment variable in Python?

The os.environ object in Python is used to retrieve environment variables. Although the os.environ object resembles a dictionary, it differs because only strings may be used as values, and they cannot be serialised to JSON. When it comes to referencing the os.environ object, you have a few choices:

a. Standard way

import os
os.environ['VAR_NAME']

b. Importing Environ Object

from os import environ
environ['VAR_NAME']

c. Renaming Environ to Env

from os import environ as env
env['VAR_NAME']

Depending on what should occur if an environment variable is missing, there are three different ways to access a given environment variable in Python.

2. How to set an environment variable in Python?

In Python, setting an environment variable is equivalent to setting a dictionary's key:

os.environ['TESTING'] = 'true'

The only permitted values in os.environ are strings, which distinguishes it from a typical dictionary:

os.environ['TESTING'] = True

Output:

TypeError: str expected, not bool

Although there are use cases for setting them as well, your application will typically just need to get environment variables.

For example, constructing a DB_URL environment variable on application start-up using DB_HOST, DB_PORT, DB_USER, DB_PASSWORD, and DB_NAME environment variables:

os.environ['DB_URL'] = \
    'psql://{user}:{password}@{host}:{port}/{name}'.format(user=os.environ['DB_USER'
        ], password=os.environ['DB_PASSWORD'], host=os.environ['DB_HOST'
        ], port=os.environ['DB_PORT'], name=os.environ['DB_NAME'])

Look at another illustration of setting a default value of a variable following the value of another variable:

Set DEBUG and TESTING to 'True' if ENV is 'development'.

if os.environ.get('ENV') == 'development':
    os.environ.setdefault('DEBUG', 'True') # Only set to True if DEBUG not set
    os.environ.setdefault('TESTING', 'True') # Only set to True if TESTING not set

3. How to delete an environment variable in Python?

Use the os.environ.pop function to remove a Python environment variable. For example, you might want to remove the other DB_ prefixed fields to make sure that the app can only access the database through the DB _URL that we previously gave.

Let’s see an additional example of removing an environment variable when it is no longer required:

auth_api(os.environ['API_KEY']) # Use API_KEY
os.environ.pop('API_KEY') # Delete API_KEY as it's no longer needed

4. How to retrieve list of Environment Variables?

To see every environment variable in Python, you could use the os.environ object. It denotes environment variables with their values. It provides a dictionary with the environmental variable names as the “keys” and their values as the “values”.

In the following programme, we loop through the dictionary returned by the os.environ. The key represents the name of the environment variables, and the value represents the value of the environment variables:

import os

for name, value in os.environ.items():
    print("{0}: {1}".format(name, value))

Python environment variables examples

1. No default value

Get the environment variable directly if your app crashes when it is not set:

Example 1:

print(os.environ['HOME'])

Output:

'/home/ubuntu'

Example 2:

print(os.environ['DOES_NOT_EXIST']

Output:

Will raise a KeyError exception

For instance, the application shouldn't be able to launch if a key environment variable is not declared and a default value, such as a database password, cannot be supplied. Instead of the typical KeyError exception (which doesn't explain why your app didn't launch), if you could catch the KeyError exception and display a helpful message instead, then:

import os
import sys

# Ensure all required environment variables are set
try:
    os.environ['API_KEY']
except KeyError:
    print '[error]: `API_KEY` environment variable required'
sys.exit(1)

2. With a default value

Using the os.environ.get method and passing the default value as the second parameter, you can retrieve a default value back if an environment variable is missing:

If HOSTNAME doesn't exist, presume local development and return localhost:

print(os.environ.get('HOSTNAME', 'localhost')

None is returned if the variable doesn't exist and you call os.environ.get without specifying a default value.

assert os.environ.get('NO_VAR_EXISTS') == None

3. Conditional logic if the value exists

Although you might want to know if an environment variable exists, you don't necessarily need to know its value. For instance, If the DEBUG environment variable is set, your application may enter "Debug mode."

You can confirm an environment variable's existence by itself:

if 'DEBUG' in os.environ:
	print('[info]: app is running in debug mode')
if os.environ.get('DEBUG') == 'True':
	print('[info]: app is running in debug mode')

Setting Environment Variables in Python with .env file

The number of environmental variables increases along with the size and complexity of a programme. Most of the projects experience growing difficulties when using environment variables for application configuration and secrets because there is a lack of clear and consistent strategy for how to manage them, especially when deploying to multiple environments.

A simple and better solution is to use a .env file to include all of the variables for a particular environment and use a Python library such as python-dotenv to parse the .env file and populate the os.environ object.

Install the python-dotenv library after creating and activating a new virtual environment to follow along:

1. Create

python3 -m venv ~/.virtualenvs/doppler-tutorial

2. Activate

source ~/.virtualenvs/doppler-tutorial/bin/activate

3. Install dotenv package

pip install python-dotenv

Save the following to a file with the extension .env (notice that it uses the same syntax as setting a variable in the shell):

API_KEY="357A70FF-BFAA-4C6A-8289-9831DDFB2D3D"
HOSTNAME="0.0.0.0"
PORT="8080"

Next, save the subsequent code to dotenv-test.py.

4. Renaming os.environ to env

from os import environ as envfrom dotenv import load_dotenv

load_dotenv()

print('API_KEY:  {}'.format(env['API_KEY']))
print('HOSTNAME: {}'.format(env['HOSTNAME']))
print('PORT:     {}'.format(env['PORT']))

After that, launch dotenv-test.py to verify that the environment variables are populating:

# python3 dotenv-test.py

API_KEY:  357A70FF-BFAA-4C6A-8289-9831DDFB2D3D
HOSTNAME: 0.0.0.0
PORT:     8080

Use cases of Environment Variables

It is used to store configuration information and application secrets, which your running application can access as needed.
Enhance security procedures. It safeguards private data. When there are parts of your code that you don't want others to see, like API tokens that give access to your accounts you only need to call the environment variables. If the API token is stored as an environment variable rather than directly typing it out in the file, users will be able to use your code without having to hardcode their API tokens.
The usage of environment variables also eliminates the need to update your source code if your secrets change.
Without making any code modifications, you can deploy your application in any environment and ensures that sensitive information, such as API keys, is not leaking into the source code.
When configuring your Python programme, environment variables are a wonderful method to do it without having to update the source code. Third-party API keys, network ports, database servers, and any other unique choices that your application may require to function properly are common configuration elements that are frequently supplied to applications using environment variables.
You can avoid manually updating your environment variables. For example, when running a script on someone else's computer, you don't need to manually change the "user" portion of the path; instead, you can use an environment variable that returns the user's name. This means that when utilising a script on a different system, you no longer need to remember to modify that.
You can do development deployment or stage in deployment or even production deployment with environment variables. Environment variables will keep all the secrets safe.

Final thoughts

In this blog, we looked at how to use the Python dotenv package and an.env file to manage environment variables.

Environment variables play a vital role to isolate sensitive data from your application. They assist in keeping your app's secrets safe and make it simple for you to switch between sets of secrets based on the circumstances. But taking care of them adds another duty to your plate.

Your client ids, hostnames, database names, port numbers, secrets, and other data are frequently stored in environment variables in the form of key-value pairs to keep them safe.

python-dotenv can be a very helpful tool for handling various environment variables, sensitive data and configuration choices in your Python applications and helps to avoid hard-coding sensitive information into your code.

Environment variables files are text files with the .env extension that can be used to store environmental variables. It's a quick and easy approach to save global variables that you want to utilise but don't want to be accessible to the public.

Atatus: Python Performance Monitoring

Atatus is an Application Performance Management (APM) solution that collects all requests to your Python applications without requiring you to change your source code. However, the tool does more than just keep track of your application's performance.

Monitor logs from all of your Python applications and systems into a centralized and easy-to-navigate user interface, allowing you to troubleshoot faster using Python monitoring.

We give a cost-effective, scalable method to centralized Python logging, so you can obtain total insight across your complex architecture. To cut through the noise and focus on the key events that matter, you can search the logs by hostname, service, source, messages, and more. When you can correlate log events with APM slow traces and errors, troubleshooting becomes easy.

Try your 14-day free trial of Atatus.