Ansible and Python 3¶

Background¶

One of the most essential things to decide upon for porting code to Python-3 is what string model to use. Strings can be an array of bytes (like in C) or they can be an array of text. Text is what we think of as letters, digits, numbers, other printable symbols, and a small number of unprintable “symbols” (control codes).

In Python-2, the two types for these (str for bytes and unicode for text) are often used interchangeably. When dealing only with ASCII characters, the strings can be combined, compared, and converted from one type to another automatically. When non-ASCII characters are introduced, Python starts throwing exceptions due to not knowing what encoding the non-ASCII characters should be in.

Python-3 changes this behavior by making the separation between bytes (bytes) and text (str) more strict. Python will throw an exception when trying to combine and compare the two types. The programmer has to explicitly convert from one type to the other to mix values from each.

This change makes it immediately apparent to the programmer when code is mixing the types inappropriately, rather than working until one of their users causes an exception by entering non-ASCII input. However, it forces the programmer to proactively define a strategy for working with strings in their program so that they don’t mix text and byte strings unintentionally.

Unicode Sandwich¶

In controller-side code we use a strategy known as the Unicode Sandwich (named after Python-2’s unicode text type). For Unicode Sandwich we know that at the border of our code and the outside world (for example, file and network IO, environment variables, and some library calls) we are going to receive bytes. We need to transform these bytes into text and use that throughout the internal portions of our code. When we have to send those strings back out to the outside world we first convert the text back into bytes. To visualize this, imagine a ‘sandwich’ consisting of a top and bottom layer of bytes, a layer of conversion between, and all text type in the center.

Common Borders¶

This is a partial list of places where we have to convert to and from bytes. It’s not exhaustive but gives you an idea of where to watch for problems.

from ansible.module_utils._text import to_text

with open('filename-with-utf8-data.txt', 'rb') as my_file:
    b_data = my_file.read()
    try:
        data = to_text(b_data, errors='surrogate_or_strict')
    except UnicodeError:
        # Handle the exception gracefully -- usually by displaying a good
        # user-centric error message that can be traced back to this piece
        # of code.
        pass

from ansible.module_utils._text import to_bytes

with open('filename.txt', 'wb') as my_file:
    my_file.write(to_bytes(some_text_string))

import os.path

from ansible.module_utils._text import to_bytes

filename = u'/var/tmp/くらとみ.txt'
f = open(to_bytes(filename), 'wb')
mtime = os.path.getmtime(to_bytes(filename))
b_filename = os.path.expandvars(to_bytes(filename))
if os.path.exists(to_bytes(filename)):
    pass

import os.path

os.path.join(u'/var/tmp/café', u'くらとみ')
os.path.split(u'/var/tmp/café/くらとみ')

Forwards Compatibility Boilerplate¶

Use the following boilerplate code at the top of all controller-side modules to make certain constructs act the same way on Python-2 and Python-3:

# Make coding more python3-ish
from __future__ import (absolute_import, division, print_function)
__metaclass__ = type

__metaclass__ = type makes all classes defined in the file into new-style classes without explicitly inheriting from object.

The __future__ imports do the following:

Prefix byte strings with “b_”¶

Since mixing text and bytes types leads to tracebacks we want to be clear about what variables hold text and what variables hold bytes. We do this by prefixing any variable holding bytes with b_. For instance:

filename = u'/var/tmp/café.txt'
b_filename = to_bytes(filename)
with open(b_filename) as f:
    data = f.read()

We do not prefix the text strings instead because we only operate on byte strings at the borders, so there are fewer variables that need bytes than text.

Bundled six¶

The third-party python-six library exists to help projects create code that runs on both Python-2 and Python-3. Ansible includes a version of the library in module_utils so that other modules can use it without requiring that it is installed on the remote system. To make use of it, import it like this:

from ansible.module_utils import six

Exceptions¶

In order for code to function on Python-2.6+ and Python-3, use the new exception-catching syntax which uses the as keyword:

try:
    a = 2/0
except ValueError as e:
    module.fail_json(msg="Tried to divide by zero: %s" % e)

Do not use the following syntax as it will fail on every version of Python-3:

try:
    a = 2/0
except ValueError, e:
    module.fail_json(msg="Tried to divide by zero: %s" % e)

Octal numbers¶

In Python-2.x, octal literals could be specified as 0755. In Python-3, octals must be specified as 0o755.

String formatting¶

# Does not work in Python-2.6!
new_string = "Dear {}, Welcome to {}".format(username, location)

# Use this instead
new_string = "Dear {0}, Welcome to {1}".format(username, location)

b_command_line = b'ansible-playbook --become-user %s -K %s' % (user, playbook_file)

Python-2.4 Compatible Exception Syntax¶

Until Ansible-2.4, modules needed to be compatible with Python-2.4 as well. Python-2.4 did not understand the new exception-catching syntax so we had to write a compatibility function that could work with both Python-2 and Python-3. You may still see this used in some modules:

from ansible.module_utils.pycompat24 import get_exception

try:
    a = 2/0
except ValueError:
    e = get_exception()
    module.fail_json(msg="Tried to divide by zero: %s" % e)

Unless a change is going to be backported to Ansible-2.3, you should not have to use this in new code.

Python 2.4 octal workaround¶

Before Ansible-2.4, modules had to be compatible with Python-2.4. Python-2.4 did not understand the new syntax for octal literals so we used the following workaround to specify octal values:

# Can't use 0755 on Python-3 and can't use 0o755 on Python-2.4
EXECUTABLE_PERMS = int('0755', 8)

Unless a change is going to be backported to Ansible-2.3, you should not have to use this in new code.