awesome-python/website/data/category_intros/text-processing.md

Need a Python text processing library? charset-normalizer reads text in unknown encodings, RapidFuzz does fuzzy matching, and pyparsing builds parsers.

How to choose:

• Bytes in an unknown encoding: charset-normalizer
• Fuzzy matching against many strings: RapidFuzz
• Parsers for your own grammar: pyparsing, or parsy for small languages
• Text garbled by a wrong decode (mojibake): ftfy
• Diffs, or close matches without a dependency: difflib
• ASCII-art banners: pyfiglet
• Translations, plus localized dates and numbers: Babel
• Syntax highlighting: Pygments
• Splitting and formatting SQL: sqlparse
• Phone numbers: phonenumbers
• URL slugs: python-slugify, or Unidecode for ASCII transliteration alone
• Short public IDs: shortuuid, or Sqids to encode integer IDs

Decode with the encoding you know before you guess one. ftfy's docs say to assume UTF-8 until proven otherwise, and charset-normalizer's FAQ calls detection the last resort. For bytes you have no clue about, charset-normalizer is the detector requests installs. Call from_bytes() or from_path(), then .best() to get the most probable result. Its detect() is backward compatible with chardet's, so switching is one import. If you stay on chardet, read large files and streams with UniversalDetector.

ftfy fixes text that was already decoded wrong, and it doesn't take bytes, so it's no replacement for a detector. Call ftfy.fix_text(), the function you'll use most. Every fix is on by default, so check which ones fit your text.

RapidFuzz is a fast string matching library with a C++ core. To compare one string with a list, use the process module's process.extractOne() or process.cdist(), which is faster than calling the scorers yourself. It doesn't lowercase or strip punctuation for you: pass processor=utils.default_process when case and punctuation shouldn't count.

difflib ships with Python, so it works where you can't add a dependency. get_close_matches() returns the good enough matches for a word, and its diffs come out in unified, context, or HTML format.

pyparsing lets you build the grammar in Python code instead of regular expressions. Since a grammar can accept invalid input, its docs say to use it on input you assume is well-formatted. Its best practices say to write the grammar out in BNF first. When the grammar is recursive, call enable_packrat() right after the import, and give results names to the fields you read back.

parsy combines small parsers into bigger ones. Its docs say it excels at small languages and is easy to read, but to look elsewhere when you need speed or good error messages. For more complex parsers, the @generate decorator is both more readable and more powerful.

Babel does two jobs: gettext message catalogs, and CLDR locale data for localized dates, numbers, and names. Run the catalog steps with the pybabel command: extract, init, update, and compile. Keep times in UTC, and convert to the user's time zone only for input and display.

Pygments turns code into HTML, LaTeX, ANSI, and more, as a command-line tool or a library. For HTML, it writes CSS classes instead of inline styles, so generate the stylesheet with HtmlFormatter().get_style_defs(). Pick the lexer by name or file name, and guess it only when you don't know the language. Pygments doesn't guarantee how long it runs, so on user input, run it with a short timeout and cap how many run at once.

sqlparse is a non-validating SQL parser: it splits scripts into statements, formats them, and walks their tokens, without assuming a SQL dialect. Use split(), format(), and parse(). On SQL from untrusted sources, keep its grouping limits as they are.

phonenumbers is a Python port of Google's libphonenumber. Pass parse() the region the number was dialed from, unless it's in E.164 format. Then check it's possible and valid with is_possible_number() and is_valid_number().

python-slugify makes URL slugs, and Unidecode turns Unicode text into ASCII. Their licenses differ. Unidecode is GPL. python-slugify's own code is MIT, and by default it runs on text-unidecode, which offers the Artistic license or GPL. But python-slugify switches to Unidecode whenever it's installed.

shortuuid turns UUIDs into short IDs for users to see, and leaves out look-alike characters like l, 1, I, O, and 0. Sqids turns database keys and other integers into short IDs. Anyone can decode them back into numbers, so keep them away from sensitive data and user IDs. To check an ID is the canonical one, re-encode the decoded numbers and compare.

Store what these libraries generate, or pin their versions, since slugs and IDs can change between releases. python-slugify says to pin the package and its backend, and Unidecode says to store each slug once or lock the version. Sqids says to pass your own blocklist, even one identical to the default.