By Matt Schwager

Deserializing, decoding, and processing untrusted input are telltale signs that your project would benefit from fuzzing. Yes, even Python projects. Fuzzing helps reduce bugs in high-assurance software developed in all programming languages. Fortunately for the Python ecosystem, Google has released Atheris, a coverage-guided fuzzer for both pure Python code and Python C extensions. When it comes to Python projects, Atheris is really the only game in town if you’re looking for a mature fuzzer. Fuzzing pure Python code typically uncovers unexpected exceptions, which can ultimately lead to denial of service. Fuzzing Python C extensions may uncover memory errors, data races, undefined behavior, and other classes of bugs. Side effects include: memory corruption, remote code execution, and, more generally, all the headaches we’ve come to know and love about C. This post will focus on fuzzing Python C extensions.

We’ll walk you through using Atheris to fuzz Python C extensions, adding a Python project to OSS-Fuzz, and setting up continuous fuzzing through OSS-Fuzz’s integrated CIFuzz tool. OSS-Fuzz is Google’s continuous fuzzing service for open-source projects, making it a valuable tool for open-source developers; as of August 2023, it has helped find and fix over 10,000 vulnerabilities and 36,000 bugs. We will target the cbor2 Python library in our fuzzing campaign. This library is the perfect target because it performs serialization and deserialization of a JSON-like, binary format and has an optional C extension implementation for improved performance. Additionally, Concise Binary Object Representation (CBOR) is used heavily within the blockchain community, which tends to have high assurance and security requirements.

In the end, we found multiple memory corruption bugs in cbor2 that could become security vulnerabilities under the right circumstances.

Fuzzing Python C extensions

Under the hood, Atheris uses libFuzzer to perform its fuzzing. Since libFuzzer is built on top of LLVM and Clang, we will need a Clang installation to fuzz our target. To simplify the installation process, I wrote a Dockerfile to package up all the necessary components into a single Docker image. This creates a repeatable process for fuzzing the current target and an easily extensible artifact for fuzzing future targets. The resulting Docker image includes a Python fuzzing harness to initiate the fuzzing process.

First, we’ll discuss some interesting parts of this Dockerfile, then we’ll investigate the fuzz.py fuzzing harness, and finally we’ll build and run the Docker image and find some memory corruption bugs!

Fuzzing environment

Dockerfiles are a great way to create a self-documenting, reproducible environment. Since fuzzing can often be more art than science, this section will also include some discussion on interesting and non-obvious bits in the Dockerfile. The following Dockerfile was used to fuzz cbor2:

FROM debian:12-slim

RUN apt update && apt install -y \
    git \
    python3-full \
    python3-pip \
    wget \
    xz-utils \
    && rm -rf /var/lib/apt/lists/*

RUN python3 --version

ENV APP_DIR "/app"
ENV CLANG_DIR "$APP_DIR/clang"
RUN mkdir $APP_DIR
RUN mkdir $CLANG_DIR
WORKDIR $APP_DIR

ENV VIRTUAL_ENV "/opt/venv"
RUN python3 -m venv $VIRTUAL_ENV
ENV PATH "$VIRTUAL_ENV/bin:$PATH"

ARG CLANG_URL=https://github.com/llvm/llvm-project/releases/download/llvmorg-17.0.6/clang+llvm-17.0.6-aarch64-linux-gnu.tar.xz
ARG CLANG_CHECKSUM=6dd62762285326f223f40b8e4f2864b5c372de3f7de0731cb7cd55ca5287b75a

ENV CLANG_FILE clang.tar.xz
RUN wget -q -O $CLANG_FILE $CLANG_URL && \
    echo "$CLANG_CHECKSUM  $CLANG_FILE" | sha256sum -c - && \
    tar xf $CLANG_FILE -C $CLANG_DIR --strip-components 1 && \
    rm $CLANG_FILE

# https://github.com/google/atheris#building-from-source
RUN LIBFUZZER_LIB=$($CLANG_DIR/bin/clang -print-file-name=libclang_rt.fuzzer_no_main.a) \
    python3 -m pip install --no-binary atheris atheris

# https://github.com/google/atheris/blob/master/native_extension_fuzzing.md#step-1-compiling-your-extension
ENV CC "$CLANG_DIR/bin/clang"
ENV CFLAGS "-fsanitize=address,undefined,fuzzer-no-link"
ENV CXX "$CLANG_DIR/bin/clang++"
ENV CXXFLAGS "-fsanitize=address,undefined,fuzzer-no-link"
ENV LDSHARED "$CLANG_DIR/bin/clang -shared"

ARG BRANCH=master

# https://github.com/agronholm/cbor2
ENV CBOR2_BUILD_C_EXTENSION "1"
RUN git clone --branch $BRANCH https://github.com/agronholm/cbor2.git
RUN python3 -m pip install cbor2/

# Allow Atheris to find fuzzer sanitizer shared libs
# https://github.com/google/atheris/blob/master/native_extension_fuzzing.md#option-a-sanitizerlibfuzzer-preloads
ENV LD_PRELOAD "$VIRTUAL_ENV/lib/python3.11/site-packages/asan_with_fuzzer.so"

# Subject to change by upstream, but it's just a sanity check
RUN nm $(python3 -c "import _cbor2; print(_cbor2.__file__)") | grep asan \
    && echo "Found ASAN" \
    || echo "Missing ASAN"

# 1. Skip allocation failures and memory leaks for now, they are common, and low impact (DoS)
# 2. https://github.com/google/atheris/blob/master/native_extension_fuzzing.md#leak-detection
# 3. Provide the symbolizer to turn virtual addresses to file/line locations
ENV ASAN_OPTIONS "allocator_may_return_null=1,detect_leaks=0,external_symbolizer_path=$CLANG_DIR/bin/llvm-symbolizer"

COPY fuzz.py fuzz.py

ENTRYPOINT ["python3", "fuzz.py"]
CMD ["-help=1"]

The following bits of the Dockerfile are relevant for customizations or future projects and are worth discussing further:

  1. Installing Clang from the llvm-project repository
  2. Customizing the image at build-time using Docker build arguments (e.g., ARG)
  3. Installing the cbor2 project
  4. Sanity checking the compiled cbor2 C extension for AddressSanitizer (ASan) symbols using nm
  5. Using ASAN_OPTIONS to customize the fuzzing process

First, installing Clang from the llvm-project repository:

ENV APP_DIR "/app"
ENV CLANG_DIR "$APP_DIR/clang"
...
RUN mkdir $CLANG_DIR
...
ARG CLANG_URL=https://github.com/llvm/llvm-project/releases/download/llvmorg-17.0.6/clang+llvm-17.0.6-aarch64-linux-gnu.tar.xz
ARG CLANG_CHECKSUM=6dd62762285326f223f40b8e4f2864b5c372de3f7de0731cb7cd55ca5287b75a
...
ENV CLANG_FILE clang.tar.xz
RUN wget -q -O $CLANG_FILE $CLANG_URL && \
    echo "$CLANG_CHECKSUM  $CLANG_FILE" | sha256sum -c - && \
    tar xf $CLANG_FILE -C $CLANG_DIR --strip-components 1 && \
    rm $CLANG_FILE

This code installs the 17.0.6-aarch64-linux-gnu tarball of Clang. There is nothing particularly special about this tarball other than the fact that it is built for AArch64 and Linux. If you are running this Docker container on a different architecture, you will need to use the corresponding release tarball. You can then specify the CLANG_URL and CLANG_CHECKSUM build arguments as necessary or simply modify the Dockerfile according to your system’s requirements.

The Dockerfile also provides a BRANCH build argument. This allows the builder to specify a Git branch or tag that they would like to fuzz against. For example, if you’re working on a pull request and want to fuzz its corresponding branch, you can use this build argument to do so.

Next up, installing the cbor2 project:

ENV CBOR2_BUILD_C_EXTENSION "1"
RUN git clone --branch $BRANCH https://github.com/agronholm/cbor2.git
RUN python3 -m pip install cbor2/

This installs the cbor2 package from GitHub rather than from PyPI. This is necessary because we need to compile the underlying C extension. We could install the package from the PyPI source distribution, but using Git provides us more control over which branch, tag, or commit we install.

The CBOR2_BUILD_C_EXTENSION environment variable instructs setup.py to ensure the C extension is built:

 30    cpython = platform.python_implementation() == "CPython"
 31    windows = sys.platform.startswith("win")
 32    use_c_ext = os.environ.get("CBOR2_BUILD_C_EXTENSION", None)
 33    if use_c_ext == "1":
 34        build_c_ext = True
 35    elif use_c_ext == "0":
 36        build_c_ext = False
 37    else:
 38        build_c_ext = cpython and (windows or check_libc())

The environment flag for building the C extension (setup.py#30–38)

This is a common pattern for Python packages with C extensions. Investigating a project’s setup.py is a great way to better understand how a C extension is built. For more information, see the setuptools documentation on building extension modules.

On to sanity checking the compiled C extension:

RUN nm $(python3 -c "import _cbor2; print(_cbor2.__file__)") | grep asan \
    && echo "Found ASAN" \
    || echo "Missing ASAN"

This command searches the compiled C extension symbol table for ASan symbols. If they exist, then we know the C extension was compiled correctly. It is interesting to note that the __file__ attribute also works for shared objects in Python and thus enables this check:

$ python3 -c "import _cbor2; print(_cbor2.__file__)"
/opt/venv/lib/python3.11/site-packages/_cbor2.cpython-311-aarch64-linux-gnu.so

Finally, let’s dig into ASAN_OPTIONS:

ENV ASAN_OPTIONS "allocator_may_return_null=1,detect_leaks=0,external_symbolizer_path=$CLANG_DIR/bin/llvm-symbolizer"

We are specifying three options:

  1. allocator_may_return_null=1: We’re disabling this check because fuzzing runs were producing Python MemoryError exceptions. We’re only looking for C memory corruption bugs, not Python exceptions.
  2. detect_leaks=0: This option is recommended by the Atheris documentation.
  3. external_symbolizer_path=$CLANG_DIR/bin/llvm-symbolizer: This enables the LLVM symbolizer to turn virtual addresses to file/line locations in fuzzing output.

You can find the full list of ASan sanitizer flags and common sanitizer options in Google’s sanitizers repository.

Fuzzing harness

The fuzzing harness used for cbor2 was largely inspired by the harness used by ujson in Google’s oss-fuzz repository. There are hundreds of projects being fuzzed in this repository. Reading through their fuzzing harnesses is a great way to gather ideas for your fuzzing project.

The following is the Python code used as the fuzzing harness:

#!/usr/bin/python3

import sys
import atheris

# _cbor2 ensures the C library is imported
from _cbor2 import loads

def test_one_input(data: bytes):
    try:
        loads(data)
    except Exception:
        # We're searching for memory corruption, not Python exceptions
        pass

def main():
    atheris.Setup(sys.argv, test_one_input)
    atheris.Fuzz()

if __name__ == "__main__":
    main()

Remember, we are fuzzing only the C extension, not the Python code. Two features of the harness enable that behavior: importing _cbor2 instead of cbor2, and the try/except block around the loads call. Looking again at setup.py, we see that _cbor2 is the Python module name for the C extension:

 47    if build_c_ext:
 48        _cbor2 = Extension(
 49            "_cbor2",
 50            # math.h routines are built-in to MSVCRT
 51            libraries=["m"] if not windows else [],
 52            extra_compile_args=["-std=c99"] + gnu_flag,
 53            sources=[
 54                "source/module.c",
 55                "source/encoder.c",
 56                "source/decoder.c",
 57                "source/tags.c",
 58                "source/halffloat.c",
 59            ],
 60            optional=True,
 61        )
 62        kwargs = {"ext_modules": [_cbor2]}
 63    else:
 64        kwargs = {}

The _cbor2 Python module name (setup.py#47–64)

That is how we know to import _cbor2 instead of cbor2. In addition to the import, the try/except block effectively ignores crashes caused by Python exceptions.

With the fuzzing environment provided by the Docker image and the fuzzing harness provided by the Python code, we are ready to do some fuzzing!

Running the fuzzer

First, copy the Dockerfile and Python code to files named Dockerfile and fuzz.py, respectively. You can then build the Docker image with the following command:

$ docker build --build-arg BRANCH=5.5.1 -t cbor2-fuzz -f Dockerfile

Note that the APT packages and Clang installation require large downloads, so the build may take a while. Since version 5.5.1 was the latest cbor2 release when these bugs were found, we are building against that Git tag to reproduce the crashes. When the build is done, you can start the fuzzing process with the following command:

$ docker run -v $(pwd):/tmp/output/ cbor2-fuzz -artifact_prefix=/tmp/output/

Specifying /tmp/output as both a Docker volume and the libFuzzer artifact_prefix will cause any crash output files to persist to the host’s filesystem rather than the container’s ephemeral filesystem. See the libFuzzer options documentation for more information on flags that can be passed at runtime.

Running the fuzzer should quickly produce the following crash:

/usr/include/python3.11/object.h:537:15: runtime error: member access within null pointer of type 'PyObject' (aka 'struct _object')
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior /usr/include/python3.11/object.h:537:15 in 
AddressSanitizer:DEADLYSIGNAL
=================================================================
==1==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000000 (pc 0xffff921a94b4 bp 0xffffe8dc8ce0 sp 0xffffe8dc8ca0 T0)
==1==The signal is caused by a READ memory access.
==1==Hint: address points to the zero page.
    #0 0xffff921a94b4 in Py_DECREF /usr/include/python3.11/object.h:537:9
    #1 0xffff921a94b4 in decode_definite_string /app/cbor2/source/decoder.c:653:9
    #2 0xffff921a94b4 in decode_string /app/cbor2/source/decoder.c:718:15
    #3 0xffff921a5cc8 in decode /app/cbor2/source/decoder.c:1735:27
    #4 0xffff921b1d98 in CBORDecoder_decode_stringref_ns /app/cbor2/source/decoder.c:1456:15
    #5 0xffff921ab90c in decode_semantic /app/cbor2/source/decoder.c:973:31
    #6 0xffff921a5d48 in decode /app/cbor2/source/decoder.c:1738:27
    #7 0xffff921aac90 in decode_map /app/cbor2/source/decoder.c:909:27
    #8 0xffff921a5d28 in decode /app/cbor2/source/decoder.c:1737:27
    #9 0xffff921d4e28 in CBOR2_load /app/cbor2/source/module.c:318:19
    #10 0xffff921d4e28 in CBOR2_loads /app/cbor2/source/module.c:367:19
    ...
==1==ABORTING
MS: 1 ChangeByte-; base unit: 096adbe21e6ccdcdaf3b466eae0eecc042a4ce48
0xa9,0xd9,0x1,0x0,0x67,0x0,0xfa,0xfa,0x0,0x0,0x4,0x4,
\251\331\001\000g\000\372\372\000\000\004\004
artifact_prefix='/tmp/output/'; Test unit written to /tmp/output/crash-092ce4a82026ba5ca35d4ee4ef5c9ba41623d61d
Base64: qdkBAGcA+voAAAQE

The output gives us the full stack trace and a crash file to reproduce the issue:

$ python -m cbor2.tool -p crash-092ce4a82026ba5ca35d4ee4ef5c9ba41623d61d 
Segmentation fault: 11

The crash happens in the Py_DECREF call in decode_definite_string:

 640    PyObject *ret = NULL;
 641    char *buf;
 642    
 643    buf = PyMem_Malloc(length);
 644    if (!buf)
 645       return PyErr_NoMemory();
 646    
 647    if (fp_read(self, buf, length) == 0)
 648       ret = PyUnicode_DecodeUTF8(
 649               buf, length, PyBytes_AS_STRING(self->str_errors));
 650    PyMem_Free(buf);
 651    
 652    if (string_namespace_add(self, ret, length) == -1) {
 653       Py_DECREF(ret);
 654       return NULL;
 655    }
 656    return ret;

The Py_DECREF call (source/decoder.c#640–656)

A NULL pointer dereference in the Python standard library produces the crash. Since the Py_DECREF documentation states that the passed object must not be NULL, the cbor2 developers fixed this bug by adding code that will detect a NULL pointer and return an error before Py_DECREF is reached.

Integrating a project into OSS-Fuzz

Google created OSS-Fuzz to improve the state of security for open-source projects. The service describes itself as “… a free service that runs fuzzers for open source projects and privately alerts developers to the bugs detected.” Integrating a project into OSS-Fuzz is a straightforward process. However, be aware that acceptance into OSS-Fuzz is ultimately at the discretion of the OSS-Fuzz team. There is no guarantee that a project will be accepted. OSS-Fuzz gives each new project proposal a criticality score and uses this value to determine if a project should be accepted.

Integrating a project into OSS-Fuzz requires four files:

  1. project.yaml: This file contains metadata about your project like contact information, repository location, programming language, and fuzzing engine.
  2. Dockerfile: This file clones your project and copies any necessary fuzzing resources like corpora or dictionaries into a Docker image. OSS-Fuzz will then run the Docker image as part of the fuzzing process.
  3. build.sh: This file installs your project and any of its dependencies into the Docker image fuzzing environment.
  4. A fuzzing harness file: This initiates the fuzzing process against a target. For example, to fuzz a specific Python function, the harness would be a Python script that initializes the fuzzing process with the target function.

If you would like to learn more about any of these files and their respective options, see the OSS-Fuzz documentation on setting up a new project. Once your project has been accepted to OSS-Fuzz, you will be granted access to the ClusterFuzz web interface, which provides access to crashes, coverage information, and fuzzer statistics. OSS-Fuzz will then fuzz your project in the background and notify you when it produces findings.

As part of our work fuzzing the cbor2 project, we integrated it into OSS-Fuzz in this pull request: google/oss-fuzz#11444. cbor2 will now be continuously fuzzed for bugs as development proceeds. To get a better idea of what this looks like in practice, see the cbor2 project in OSS-Fuzz.

Continuous fuzzing with CIFuzz

There’s continuous, and then there’s continuous. OSS-Fuzz fuzzes your project about once a day. If you need something more continuous than that, like, say, on every commit, then you will have to reach for another tool. Fortunately, Google and the OSS-Fuzz ecosystem have you covered with CIFuzz. CIFuzz integrates into the OSS-Fuzz ecosystem to fuzz your project on every commit. It does require a project to already be accepted and integrated in OSS-Fuzz, but non-OSS-Fuzz projects can use ClusterFuzzLite.

To take our cbor2 fuzzing one step further, we added a CIFuzz job to the project’s GitHub Actions. This will fuzz the project on every commit and every pull request. Using OSS-Fuzz and CIFuzz allows for both faster fuzz feedback on proposed changes and deeper fuzz testing as part of a scheduled nightly job. The best of both worlds. Think of it like the testing pyramid: unit tests are fast and run on every commit, whereas end-to-end tests are slow and may be run only as part of a lengthier, nightly CI job.

Once your project is integrated into OSS-Fuzz, adding CIFuzz is as simple as adding a GitHub Actions workflow to your project. This workflow file specifies similar metadata as the project’s project.yaml file, information like the project programming language, libFuzzer sanitizers to use, and fuzzing duration.

You may be asking yourself, “how long should I be fuzzing my project for?” The answer often ends up being more art than science. CIFuzz’s default duration is 600 seconds, or 10 minutes. This is a great starting point. In this situation, bigger is not always better. Remember, you could be waiting for this job to complete on every commit. How long would you and your teammates like to wait for a CI job? A good rule of thumb is that continuous fuzzing on every commit should be run for minutes, not hours or days, and that scheduled, nightly fuzzing should be run for hours, or even days. Start with something reasonable and be prepared to tweak it as necessary.

As part of our work fuzzing the cbor2 project, we added a CIFuzz workflow in this pull request: agronholm/cbor2#212. This should complement the scheduled OSS-Fuzz job nicely.

Build your own trophy case with fuzzing

Fuzzing is a great testing methodology for uncovering hard-to-find bugs and security vulnerabilities. It is particularly useful for projects performing decoding or deserialization functionality or taking in untrusted input. It has a proven track record, considering AFL’s extensive trophy case, rust-fuzz’s trophy case, and OSS-Fuzz’s claim of over 10,000 security vulnerabilities and 36,000 bugs found. Fuzzing is an advanced testing methodology, so it is not the first tool you should reach for when looking to improve your project’s robustness, but it is unquestionably a useful tool when you are looking to go to the next level.

In this post, we walked you through setting up a fuzzing environment and harness for Python C extensions and then went over the process of integrating a project into OSS-Fuzz and adding a CIFuzz GitHub Actions workflow. In the end, we found some interesting memory corruption bugs in the cbor2 Python library and made the open-source software community a little bit more secure.

If you’d like to read more about our work on fuzzing, we have used its capabilities in several ways, such as fuzzing x86_64 instruction decoders, breaking the Solidity compiler with a fuzzer, and fuzzing wolfSSL with tlspuffin.

Contact us if you’re interested in custom fuzzing for your project.