From f2a18283fd8e42d86291a233367a207fae12de39 Mon Sep 17 00:00:00 2001 From: Vinta Chen Date: Fri, 16 Oct 2026 18:37:40 +0800 Subject: [PATCH] docs: add OpenCC to Natural Language Processing intro Co-Authored-By: Claude --- website/data/category_intros/natural-language-processing.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/website/data/category_intros/natural-language-processing.md b/website/data/category_intros/natural-language-processing.md index 91515d61..6a0e1a73 100644 --- a/website/data/category_intros/natural-language-processing.md +++ b/website/data/category_intros/natural-language-processing.md @@ -8,6 +8,7 @@ How to choose: - Topic models and word embeddings: gensim - Chinese word segmentation: jieba - Chinese characters to pinyin: pypinyin +- Traditional/Simplified Chinese conversion: OpenCC - Spaces between CJK text and letters or digits: pangu.py spaCy is [designed for production use](https://spacy.io/usage/facts-figures#comparison-usage): you build and train NLP pipelines, then package them to deploy. Its trained pipelines are Python packages. In an automated build, [install them with pip from a direct link](https://spacy.io/usage/models#download-pip) instead of spaCy's download command, and put that link in your requirements.txt. In a larger code base, [import the pipeline as a module](https://spacy.io/usage/models#models-loading), so a missing one raises an ImportError right away. Run your texts through `nlp.pipe` [in batches](https://spacy.io/usage/processing-pipelines#processing), and disable the components you don't need. To train your own pipeline, run [`spacy train` with one config file](https://spacy.io/usage/training#quickstart). @@ -20,6 +21,8 @@ jieba [segments Chinese text into words](https://github.com/fxsjy/jieba): accura pypinyin [matches pinyin by whole words](https://github.com/mozillazg/python-pinyin), so it handles characters with more than one reading. It also writes zhuyin (Bopomofo) and Wade-Giles. When a wrong word split gives a wrong reading, [segment the text with jieba first](https://pypinyin.readthedocs.io/zh-cn/latest/faq.html) and pass in the list of words. For readings that are still wrong, [add your own](https://pypinyin.readthedocs.io/zh-cn/latest/usage.html#custom-dict) with `load_phrases_dict()` or `load_single_dict()`. +OpenCC [converts between Traditional and Simplified Chinese, including the regional wording of Mainland China, Taiwan, and Hong Kong](https://github.com/BYVoid/OpenCC). It's dictionary-based, not a language model, so the results are predictable and it runs offline. Install it with `pip install opencc`, then [create a converter from a config file](https://github.com/BYVoid/OpenCC#python): `opencc.OpenCC('s2t.json').convert(text)`. [The config picks the direction and the variant](https://github.com/BYVoid/OpenCC#configurations-配置文件): `s2t.json` gives OpenCC's standard Traditional characters, `s2tw.json` gives Taiwan's, and `s2twp.json` also swaps in Taiwan's phrases. + pangu.py [inserts spaces between CJK characters and letters, digits, and symbols](https://github.com/vinta/pangu.py). Call `pangu.space_text()` on a string or `pangu.space_file()` on a file. From the command line, `pangu-py -c` prints the corrected text and exits with 1 when the spacing needed fixing. Check the license of the models and data, not only the library. spaCy is [MIT](https://github.com/explosion/spaCy/blob/master/LICENSE), but its [Spanish pipelines](https://spacy.io/models/es) are GPL and its [Italian ones](https://spacy.io/models/it) are for non-commercial use only. NLTK is [Apache](https://github.com/nltk/nltk/blob/develop/LICENSE.txt), and its corpora come [under various licenses](https://github.com/nltk/nltk/wiki/FAQ), each listed in its own README.