CLI Utilities in Python

Guide to building out command-line utilities in Python, using argparse, logging, and sys.

Most of my programming revolves around command-line utilities. The idea of ever having to program a GUI sounds nauseous.

When I write programs in Python, I generally start with a script that pulls down the main library and runs various functions to get the process started. The script almost always imports the logging library to configure the logger and the argparse library for argument parsing.

Using argparse

The Python Standard Library includes a module for parsing command-line arguments that is feature-rich and fairly easy to use: argparse. This won't be a complete guide to argparse so much as a rough overview for the unfamiliar.

To begin, import sys, argparse, and logging, as well as your application module, which is called wk in this example:

# Module Imports
import argparse
import logging
import sys

# Local Imports
import wk

Initialize Parser

Put the rest of the code in an if block to protect it from accidental module loads by other processes:

if __name__ == '__main__':

  parser = argparse.ArgumentParser()
  parser.set_defaults(func=wk.run_version)

There are two things that happen here. First, the argparse.ArgumentParser() method initializes the parser variable. Secondly, we set the default function for the parser as wk.run_version(). This is a function defined below to report version information for the utility.

Options

Most of my utilities use a handful of common options, like -v for verbosity and --debug to provide debugging information as I build the tool out. I set these first, before the subparsers given that the tool [OPTIONS] [COMMAND] [ARGUMENTS] module is one that seems most sensible to me.

To add options, use the add_arguments() method with a - for short options and a -- for long options.

# Options
parser.add_argument('-v', '--verbose', action='store_true',
    help="Enables verbosity for logging messages")

parser.add_arguments('-D', '--debug', action='store_true',
    help="Enables debugging information for logging messages")

This adds -v, --verbose and -D, --debug options to the utility. Both are flags, meaning that including them in the command sets the variables to True, excluding them sets the variables to False.

Subparsers

My preferred design is that the first argument that occurs in a CLI utility usually defines the specific operation you want to run. git add or git commit as common examples. In argparse the syntax to run this operation is called a subparser.

To initialize a subparser, use the add_subparsers() method:

subparsers = parser.add_subparses(title="commands")

You can now use the subparsers variable to add new subparsers. For each subparser I also run the set_defaults() method to initialize the function that should run when the subparser is called.

# Set Version Subparser
version_subparser = subparsers.add_parser('version') 
version_subparser.set_defaults(func=wk.run_version)

# Set List Subparser
list_subparser = subparsers.add_parser('list')
list_subparser.set_defaults(func=wk.list)

This creates the version and list subparsers, setting different functions to run when they're called.

Parsing Arguments

Once you've finished specifying the options, arguments and subparsers, you need to parse them into a variable to use them in your application.

args = parser.parse_args()

The args variable now contains all of the attributes set by your specification with the values passed in from the command-line.

Logging Configuration

The logging module in the Python Standard Library has a number of features built in that I pretty much completely ignore in practice. The reason is that the only sort of logging I care about involves venting to stdout.

log_level = logging.WARN
log_format = "[ %(levelname)s ]: %(message)s"
if args.debug:
    log_level = logging.DEBUG

    if args.verbose:
       log_format = "[ %(levelname)s:%(filename)s:%(lineno)s ]: %(message)s"
elif args.verbose:
    log_level = logging.INFO
else:
    log_level = logging.WARN

logging.basicConfig(format=log_format, level=log_level)

There are two variables set in this block: log_level and log_format. The utility uses log_level to specify the logging level that appears to the user. It defaults to warnings and critical messages only, but shows information messages with --verbose option and debugging information with the --debug option.

The log_message variable prints the log level and log message by default, but when the --debug and --verbose options are used together, it expands the log level to include the name of the file in which the logging message was called and the line number. That is, information useful to the developer but not very useful to the user.

Running Applications

Last bit of code in the script is the part where we call functions from the library. Generally, I also include a call to sys.exit() to avoid having to run it in the individual functions.

# Run Utility
args.func(args)

# Exit
sys.exit(0)

Remember above in the argparse code, the set_defaults() method was used repeatedly to set a func attribute matching functions in the library. Specifically wk.run_version() and wk.run_list(). These functions were set without being called, so now whichever subparser is called by the utility, its function is set as args.func(). We then pass in the args variable so that the other options are available to the program.

In the wk/__init__.py file we then add the functions for argparse to call:

# Initialize Logger
from logging import getLogger
logger = getLogger()

# Set Version
version = "0.1"

#################### RUN VERSION ##################
def run_version(args):
  """Reports the version number to stdout"""
  print(f"wk - version {version}")

###################### RUN LIST ####################
def run_list(args):
  """Runs the list operation"""
  logger.info("Called 'list' operation")
  ...

When argparse initializes itself, it is set to pull the run_%() functions in from this file, they in turn contain the operational code for the utility.