blog.robur.coop/articles/2024-02-03-python-str-repr.md

---
date: 2024-02-03
article.title: Python's `str.__repr__()`
article.description: Reimplementing Python string escaping in OCaml
tags:
  - OCaml
  - Python
  - unicode
author:
  name: Reynir Björnsson
  email: reynir@reynir.dk
  link: https://reyn.ir/
---
Sometimes software is written using whatever built-ins you find in your programming language of choice.
This is usually great!
However, it can happen that you depend on the precise semantics of those built-ins.
This can be a problem if those semantics become important to your software and you need to port it to another programming language.
This story is about Python and its `str.__repr()__` function.

The piece of software I was helping port to [OCaml][ocaml] was constructing a hash from the string representation of a tuple.
The gist of it was basically this:

```python
def get_id(x):
    id = (x.get_unique_string(), x.path, x.name)
    return myhash(str(id))
```

In other words it's a Python tuple consisting of mostly strings but also a `PosixPath` object.
The way `str()` works is it calls the `__str__()` method on the argument objects (or otherwise `repr(x)`).
For Python tuples the `__str__()` method seems to print the result of `repr()` on each elemenet separated by a comma and a space and surrounded by parenthesis.
So good so far.
If we can precisely emulate `repr()` on strings and `PosixPath` it's easy.
In the case of `PosixPath` it's really just `'PosixPath('+repr(str(path))+')'`;
so in that case it's down to `repr()` on strings - which is `str.__repr__()`,

There had been a previous attempt at this that would use OCaml's string escape functions and surround the string with single quotes (`'`).
This works for some cases, but not if the string has a double quote (`"`).
In that case OCaml would escape the double quote with a backslash (`\"`) while python would not escape it.
So a regular expression substitution was added to replace the escape sequence with just a double quote.
This pattern of finding small differences between Python and OCaml escaping had been repeated,
and eventually I decided to take a more rigorous approach to it.

## What is a string?

First of all, what is a string? In Python? And in OCaml?
In OCaml a string is just a sequence of bytes.
Any bytes, even `NUL` bytes.
There is no concept of unicode in OCaml strings.  
In Python there is the `str` type which is a sequence of Unicode code points[^python-bytes].
I can recommend reading Daniel Bünzli's [minimal introduction to Unicode][unicode-minimal].
Already here there is a significant gap in semantics between Python and OCaml.
For many practical purposes we can get away with using the OCaml `string` type and treating it as a UTF-8 encoded Unicode string.
This is what I will do as in both the Python code and the OCaml code the data being read is a UTF-8 (or often only the US ASCII subset) encoded string.

## What does a string literal look like?

### OCaml

I will not dive too deep into the details of OCaml string literals, and focus mostly on how they are escaped by the language built-ins (`String.escaped`, `Printf.printf "%S"`).
Normal printable ASCII is printed as-is.
That is, letters, numbers and other symbols except for backslash and double quote.
There are the usual escape sequences `\n`, `\t`, `\r`, `\"` and `\\`.
Any byte value can be represented with decimal notation `\032` or octal notation '\o040' or hexadecimal notation `\x20`.
The escape functions in OCaml has a preference for the decimal notation over the hexadecimal notation.
Finally I also want to mention the Unicode code point escape sequence `\u{3bb}` which represents the UTF-8 encoding of U+3BB.
While the escape functions do not use it, it will become handy later on.
Illegal escape sequences (escape sequences that are not recognized) will emit a warning but otherwise result in the escape sequence as-is.
It is common to compile OCaml programs with warnings-as-errors, however.

### Python

Python has a number of different string literals and string-like literals.
They all use single quote or double quote to delimit the string (or string-like) literals.
There is a preference towards single quotes in `str.__repr__()`.
You can also triple the quotes if you like to write a string that uses a lot of both quote characters.
This format is not used by `str.__repr__()` so I will not cover it further, but you can read about it in the [Python reference manual](https://docs.python.org/3/reference/lexical_analysis.html#strings).
The string literal can optionally have a prefix character that modifies what type the string literal is and how its content is interpreted.

The `r`-prefixed strings are called *raw strings*.
That means backslash escape sequences are not interpreted.
In my experiments they seem to be quasi-interpreted, however!
The string `r"\"` is considered unterminated!
But `r"\""` is fine as is interpreted as `'\\"'`[^raw-escape-example].
Why this is the case I have not found a good explanation for.

The `b`-prefixed strings are `bytes` literals.
This is close to OCaml strings.

Finally there are the unprefixed strings which are `str` literals.
These are the ones we are most interested in.
They use the usual escape `\[ntr"]` we know from OCaml as well as `\'`.
`\032` is **octal** notation and `\x20` is hexadecimal notation.
There is as far as I know **no** decimal notation.
The output of `str.__repr__()` uses the hexadecimal notation over the octal notation.
As Python strings are Unicode code point sequences we need more than two hexadecimal digits to be able to represent all valid "characters".
Thus there are the longer `\u0032` and the longest `\U00000032`.

## Intermezzo

While studying Python string literals I discovered several odd corners of the syntax and semantics besides the raw string quasi-escape sequence mentioned earlier.
One fact is that Python doesn't have a separate character or Unicode code point type.
Instead, a character is a one element string.
This leads to some interesting indexing shenanigans: `"a"[0][0][0] == "a"`.
Furthermore, strings separated by spaces only are treated as one single concatenated string: `"a" "b" "c" == "abc"`.
These two combined makes it possible to write this unusual snippet: `"a" "b" "c"[0] == "a"`!
For byte sequences, or `b`-prefixed strings, things are different.
Indexing a bytes object returns the integer value of that byte (or character):

```python
>>> b"a"[0]
97
>>> b"a"[0][0]
<stdin>:1: SyntaxWarning: 'int' object is not subscriptable; perhaps you missed a comma?
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'int' object is not subscriptable
```

For strings `\x32` can be said to be shorthand for `"\u0032"` (or `"\u00000032"`).
But for bytes `"\x32" != "\u0032"`!
Why is this?!
Well, bytes is a byte sequence and `b"\u0032"` is not interpreted as an escape sequence and is instead **silently** treated as `b"\\u0032"`!
Writing `"\xff".encode()` which encodes the string `"\xff"` to UTF-8 is **not** the same as `b"\xff"`.
The bytes `"\xff"` consist of a single byte with decimal value 255,
and the Unicode wizards reading will know that the Unicode code point 255 (or U+FF) is encoded in two bytes in UTF-8.

## Where is the Python code?

Finding the implementation of `str.__repr__()` turned out to not be so easy.
In the end I asked on the Internet and got a link to [cpython's `Objects/unicodeobject.c`][unicodeobject.c].
And holy cow!
That's some 160 lines of C code with two loops, a switch statement and I don't know how many chained and nested if statements!
Meanwhile the OCaml implementation is a much less daunting 52 lines of which about a fifth is a long comment.
It also has two loops which each contain one much more tame match expression (roughly a C switch statement).
In both cases they first loop over the string to compute the size of the output string.
The Python implementation also counts the number of double quotes and single quotes as well as the highest code point value.
The latter I'm not sure why they do, but my guess it's so they can choose an efficient internal representation.
Then the Python code decides what quote character to use with the following algorithm:  
Does the string contain single quotes but no double quotes? Then use double quotes. Otherwise use single quotes.
Then the output size estimate is adjusted with the number of backslashes to escape the quote character chosen and the two quotes surrounding the string.

Already here it's clear that a regular expression substitution is not enough by itself to fix OCaml escaping to be Python escaping.
My first step then was to implement the algorithm only for US ASCII.
This is simpler as we don't have to worry much about Unicode, and I could implement it relatively quickly.
The first 32 characters and the last US ASCII character (DEL or `\x7f`) are considered non-printable and must be escaped.
I then wrote some simple tests by hand.
Then I discovered the OCaml [py][ocaml-py] library which provides bindings to Python from OCaml.
Great! This I can use to test my implementation against Python!

## How about Unicode?

For the non-ascii characters (or code points rather) they are either considered *printable* or *non-printable*.
For now let's look at what that means for the output.
A printable character is copied as-is.
That is, there is no escaping done.
Non-printable characters must be escaped, and python wil use `\xHH`, `\uHHHH` or `\UHHHHHHHH` depending on how many hexadecimal digits are necessary to represent the code point.
That is, the latin-1 subset of ASCII (`0x80`-`0xff`) can be represented using `\xHH` and neither `\u00HH` nor `\U000000HH` will be used etc.

### What is a printable Unicode character?

In the cpython [function][unicodeobject.c] mentioned earlier they use the function `Py_UNICODE_ISPRINTABLE`.
I had a local clone of the cpython git repository where I ran `git grep Py_UNICODE_ISPRINTABLE` to find information about it.
In [unicode.rst][unicode.rst-isprintable] I found a documentation string for the function that describes it to return false if the character is nonprintable with the definition of nonprintable as the code point being in the categories "Other" or "Separator" in the Unicode character database **with the exception of ASCII space** (U+20 or ` `).

What are those "Other" and "Separator" categories?
Further searching for the function definition we find in [`Include/cpython/unicodeobject.h`][unicodeobject.h-isprintable] the definition.
Well, we find `#define Py_UNICODE_ISPRINTABLE(ch) _PyUnicode_IsPrintable(ch)`.
On to `git grep _PyUnicode_IsPrintable` then.
That function is defined in [`Objects/unicodectype.c`][unicodectype.c-isprintable].

```C
/* Returns 1 for Unicode characters to be hex-escaped when repr()ed,
   0 otherwise.
   All characters except those characters defined in the Unicode character
   database as following categories are considered printable.
      * Cc (Other, Control)
      * Cf (Other, Format)
      * Cs (Other, Surrogate)
      * Co (Other, Private Use)
      * Cn (Other, Not Assigned)
      * Zl Separator, Line ('\u2028', LINE SEPARATOR)
      * Zp Separator, Paragraph ('\u2029', PARAGRAPH SEPARATOR)
      * Zs (Separator, Space) other than ASCII space('\x20').
*/
int _PyUnicode_IsPrintable(Py_UCS4 ch)
{
    const _PyUnicode_TypeRecord *ctype = gettyperecord(ch);

    return (ctype->flags & PRINTABLE_MASK) != 0;
}
```

Ok, now we're getting close to something.
Searching for `PRINTABLE_MASK` we find in [`Tools/unicode/makeunicodedata.py`][makeunicodedata.py-printable-mask] the following line of code:

```Python
if char == ord(" ") or category[0] not in ("C", "Z"):
    flags |= PRINTABLE_MASK
```

So the algorithm is really if the character is a space character or if its Unicode general category doesn't start with a `C` or `Z`.
This can be implemented in OCaml using the uucp library as follows:

```OCaml
let py_unicode_isprintable uchar =
  (* {[if char == ord(" ") or category[0] not in ("C", "Z"):
           flags |= PRINTABLE_MASK]} *)
  Uchar.equal uchar (Uchar.of_char ' ')
  ||
  let gc = Uucp.Gc.general_category uchar in
  (* Not those categories starting with 'C' or 'Z' *)
  match gc with
  | `Cc | `Cf | `Cn | `Co | `Cs | `Zl | `Zp | `Zs -> false
  | `Ll | `Lm | `Lo | `Lt | `Lu | `Mc | `Me | `Mn | `Nd | `Nl | `No | `Pc | `Pd
  | `Pe | `Pf | `Pi | `Po | `Ps | `Sc | `Sk | `Sm | `So ->
      true
```

After implementing unicode I expanded the tests to generate arbitrary OCaml strings and compare the results of calling my function and Python's `str.__repr__()` on the string.
Well, that didn't go quite well.
OCaml strings are just any byte sequence, and ocaml-py expects it to be a UTF-8 encoded string and fails on invalid UTF-8.
Then in qcheck you can "assume" a predicate which means if a predicate doesn't hold on the generated value then the test is skipped for that input.
So I implement a simple verification of UTF-8.
This is far from optimal because qcheck will generate a lot of invalid utf-8 strings.

The next test failure is some unassigned code point.
So I add to `py_unicode_isprintable` a check that the code point is assigned using ``Uucp.Age.age uchar <> `Unassigned``.

Still, qcheck found a case I hadn't considered: U+61D.
My python version (Python 3.9.2 (default, Feb 28 2021, 17:03:44)) renders this as `'\u061'` while my OCaml function prints it as-is.
In other words my implementation considers it printable while python does not.
I try to enter this Unicode character in my terminal, but nothing shows up.
Then I look it up and its name is `ARABIC END OF TEXT MARKER`.
The general category according to uucp is `` `Po ``.
So this **should** be a printable character‽

After being stumped by this for a while I get the suspicion it may be dependent on the Python version.
I am still on Debian 11 and my Python version is far from being the latest and greatest.
I ask someone with a newer Python version to write `'\u061d'` in a python session.
And 'lo! It prints something that looks like `''`!
Online I figure out how to get the unicode version compiled into Python:

```Python
>>> import unicodedata
>>> unicodedata.unidata_version
'13.0.0'
```

Aha! And with uucp we find that the unicode version that introduced U+61D to be 14.0:

```OCaml
# Uucp.Age.age (Uchar.of_int 0x61D);;
- : Uucp.Age.t = `Version (14, 0)
```

My reaction is this is seriously some ungodly mess we are in.
Not only is the code that instigated this journey highly dependent on Python-specifics - it's also dependent on the specific version of unicode and thus the version of Python!

I modify our `py_unicode_isprintable` function to take an optional `?unicode_version` argument and replace the "is this unassigned?" check with the following snippet:

```OCaml
let age = Uucp.Age.age uchar in
(match (age, unicode_version) with
| `Unassigned, _ -> false
| `Version _, None -> true
| `Version (major, minor), Some (major', minor') ->
    major < major' || (major = major' && minor <= minor'))
```

Great! I modify the test suite to first detect the unicode version python uses and then pass that version to the OCaml function.
Now I can't find anymore failing test cases!  

## Epilogue

What can we learn from this?
It is easy to say in hindsight that a different representation should have been chosen.
However, arriving at this insight takes time.
The exact behavior of `str.__repr__()` is poorly documented.
Reaching my understanding of `str.__repr__()` took hours of research and reading the C implementation.
It often doesn't seem to be worth it to spend so much time on research for a small function.
Technical debt is a real thing and often hard to predict.
Below is the output of `help(str.__repr__)`:

```Python
__repr__(self, /)
    Return repr(self)
```

Language and (standard) library designers could consider whether the slightly nicer looking strings are worth the added complexity users eventually are going to rely on - inadvertently or not.
I do think strings and bytes in Python are a bit too complex.
It is not easy to get a language lawyer[^language-lawyer] level understanding.
In my opinion it is a mistake to not at least print a warning if there are illegal escape sequences - especially considering there are escape sequences that are valid in one string literal but not another.

Unfortunately it is often the case that to get a precise specification it is necessary to look at the implementation.
For testing your implementation hand-written tests are good.
Testing against the original implementation is great, and if combined with property-based testing or fuzzing you may find failing test cases you couldn't dream up!
I certainly didn't see it coming that the output depends on the Unicode version.
As is said, testing can only show the presence of bugs, but with a, in a sense, limited domain like this function you can get pretty close to showing absence of bugs.

I enjoyed working on this.
Sure, it was frustrating and at times I discovered some ungodly properties, but it's a great feeling to study and understand something at a deeper level.
It may be the last time I need to understand Python's `str.__repr__()` this well, but if I do I now have the OCaml code and this blog post to reread.

If you are curious to read the resulting code you may find it on github at [github.com/reynir/python-str-repr](https://github.com/reynir/python-str-repr).
I have documented the code to make it more approachable and maintainable by others.
Hopefully it is not something that you need, but in case it is useful to you it is licensed under a permissive license.

If you have a project in OCaml or want to port something to OCaml and would like help from me and my colleagues at [Robur](https://robur.coop/) please [get in touch](https://robur.coop/Contact) with us and we will figure something out.
  
  
  
  




[ocaml]: https://ocaml.org/
[unicode-minimal]: https://ocaml.org/p/uucp/13.0.0/doc/unicode.html#minimal
[unicodeobject.c]: https://github.com/python/cpython/blob/963904335e579bfe39101adf3fd6a0cf705975ff/Objects/unicodeobject.c#L12245-L12405
[escaped]: https://github.com/ocaml/ocaml/blob/a51089215d5ae1187688a5b130e9f62bf50adfeb/stdlib/bytes.ml#L170-L222
[ocaml-py]: https://github.com/zshipko/ocaml-py
[unicode.rst-isprintable]: https://github.com/python/cpython/blob/963904335e579bfe39101adf3fd6a0cf705975ff/Doc/c-api/unicode.rst?plain=1#L257-L265
[makeunicodedata.py-printable-mask]: https://github.com/python/cpython/blob/963904335e579bfe39101adf3fd6a0cf705975ff/Tools/unicode/makeunicodedata.py#L450-L451
[unicodectype.c-isprintable]: https://github.com/python/cpython/blob/963904335e579bfe39101adf3fd6a0cf705975ff/Objects/unicodectype.c#L158-L163
[unicodeobject.h-isprintable]: https://github.com/python/cpython/blob/963904335e579bfe39101adf3fd6a0cf705975ff/Include/cpython/unicodeobject.h#L683

[^python-bytes]: There is as well the `bytes` type which is a byte sequence like OCaml's `string`.
The Python code in question is using `str` however.

[^raw-escape-example]: Note I use single quotes for the output. This is what Python would do. It would be equivalent to `"\\\""`.

[^language-lawyer]: [A person, usually an experienced or senior software engineer, who is intimately familiar with many or most of the numerous restrictions and features (both useful and esoteric) applicable to one or more computer programming languages. A language lawyer is distinguished by the ability to show you the five sentences scattered through a 200-plus-page manual that together imply the answer to your question “if only you had thought to look there”.](http://catb.org/jargon/html/L/language-lawyer.html)