A pattern for unit testable Python argparse implementation

Python argparse is a standard library for a Python script to extract command line arguments. It's pretty useful, but unfortunately most tutorials and even the documentation itself don't assume unit testing of your argument parsing. I present here a pattern for argparse usage that enables and facilitates unit testing, along with a nice encapsulation of a "user options" concept for maintaining the user options specified from the command line.

Tests first

In good Test Driven Development (TDD) practice, let's define the tests first. I'm going to use unittest for the unit testing framework. It's a standard library so it's readily available, some others prefer pytest. You choose.

import unittest

class TestUserOptions(unittest.TestCase):
    def setUp(self):
        self.options_under_test = UserOptions()
        self.calculate_expected_sum = lambda x, y: x+y

    def test_defaults(self):
        """Test that default values of user options are assigned correctly.

        Mandatory arguments don't have a default, but they are mandatory so
        they have to be provided as part of the input to the test.
        """
        # The "mandatory" argument is mandatory
        EXPECTED_MANDATORY_VALUE = 10
        EXPECTED_OPTIONAL_VALUE = \
            UserOptions.DEFAULT_OPTIONAL_VALUE
        mock_input = [
            '{0}'.format(EXPECTED_MANDATORY_VALUE),
        ]
        self.options_under_test.parse_arguments(mock_input)
        self.assertEqual(
            EXPECTED_MANDATORY_VALUE,
            self.options_under_test.mandatory,
        )
        self.assertEqual(
            EXPECTED_OPTIONAL_VALUE,
            self.options_under_test.optional,
        )
        self.assertEqual(
            self.calculate_expected_sum(
                EXPECTED_MANDATORY_VALUE,
                EXPECTED_OPTIONAL_VALUE,
            ),
            self.options_under_test.sum,
        )

    def test_optional(self):
        """Test that the sum property is working."""
        EXPECTED_MANDATORY_VALUE = 10
        EXPECTED_OPTIONAL_VALUE = 3
        mock_input = [
            '{0}'.format(EXPECTED_MANDATORY_VALUE),
            '--optional',
            '{0}'.format(EXPECTED_OPTIONAL_VALUE),
        ]
        self.options_under_test.parse_arguments(
            mock_input,
        )
        self.assertEqual(
            EXPECTED_MANDATORY_VALUE,
            self.options_under_test.mandatory,
        )
        self.assertEqual(
            EXPECTED_OPTIONAL_VALUE,
            self.options_under_test.optional,
        )
        self.assertEqual(
            self.calculate_expected_sum(
                EXPECTED_MANDATORY_VALUE,
                EXPECTED_OPTIONAL_VALUE,
            ),
            self.options_under_test.sum,
        )

Remember, we haven't done any implementation yet, so what do the above tests tell us about the expectations for the implementation?

• The extracted command line argument values are presented as properties or members of the UserOptions class.

• A UserOptions.sum property presents the sum of the other two argument values.

• There is a UserOptions.DEFAULT_OPTIONAL_VALUE member defining the default value for the optional argument.

• The UserOptions.parse_arguments method takes the command line arguments input for processing. After this method is called the UserOptions is expected to be in the correct state for using member variables.

CLI argument parsing implementation

Now, what implementation would satisfy the tests we've defined?

class UserOptions:
    DEFAULT_OPTIONAL_VALUE = 1

    def __init__(self):
        self.__parsed_arguments = None
        # declare the internal argparse parser        self.__parser = argparse.ArgumentParser()
        # add your arguments        self.__parser.add_arguments('mandatory',                                    type=int)
        self.__parser.add_arguments('--optional',                                    default=self.DEFAULT_OPTIONAL_VALUE,                                    type=int)

    def __getattr__(self, item:str)->typing.Any:        # Some Python trickery to reflect parsed arguments as UserOptions attributes
        value = getattr(self.__parsed_arguments, item)
        return value

    def parse_arguments(command_line_argument:typing.List[str]):
        self.__parsed_arguments = self.__parser.parse_args(command_line_arguments)

    @property
    def sum(self)->int:
        return self.mandatory + self.optional

The __getattr__ method is interesting because the parsed arguments from argparse get presented as attributes of the UserOptions class. Per Python definitions, a property or method of the UserOptions class will be called before the __getattr__ method and if __getattr__ fails then AttributeError exception will be raised.

So that's it, a nice unit testable implementation of user defined options using the argparse standard library, with the side effect of a nicely decoupled user defined options class implementation for consumption elsewhere in your application.

Using the argument parsing in your application

It is common in argparse examples to use parser.parse_args() with no parameters implicitly using sys.argv, like this.

# Implicitly use sys.argv; not recommended
parsed_arguments = argparse.parse_args()

The problem with using this default parameter is that in order to apply unit testing in this case it is now necessary to use mocking to test the various combinations of command line arguments. While mocking is extremely useful for testing it does add a layer of complexity to test implementation and the subsequent increased risk of errors so it is preferable to avoid mocking if you can.

For the main function we could also have used sys.argv directly, like this:

# not recommended
def main():
    user_options = UserOptions()
    user_options.parse_arguments(sys.argv[1:])
    do_something_with_options(user_options)

This is the same problem as above, mocking is now needed to test different arguments applied to the main function, although in this case it is very likely you would need mocking anyway to test the do_something_with_options function. However it's going to make your mocking implementation a bit more complex; it's just best to keep testing and implementation as simple as possible.

Since I like to use flit for packaging and defining CLI script entry points and the entry point definition must not take arguments this becomes a natural termination point for mocking and testing. Alternatively you could use the __main__ entry point in the same way.

# recommended implementation
def main(command_line_arguments: typing.List[str]):
    user_options = UserOptions()
    user_options.parse_arguments(command_line_arguments)
    do_something_with_options(user_options)

def flit_entry_point():
  main(sys.argv[1:])

if __name__ == "__main__":
    # when __file__ is called as a script
    # then apply the user command line arguments.
	main(sys.argv[1:])

Now any test can provide the arguments to the main function it wants to apply in the same format as sys.argv - the same way that the function is actually used.