Recently I've been working on Chinese word segmentation, starting with a baseline: dictionary-based backward maximum matching. This post introduces the most basic version.

The project is located at:

[github_widget repo="yuikns/argcv-analyzer-cn"]

As usual, it depends on my libargcv.

[github_widget repo="yuikns/argcv"]

The dictionary is based on

[github_widget repo="ling0322/webdict"]

The main code is very simple; it's in this file. The idea is to match each sentence from back to front, greedily taking the longest possible word as a token.

This method is the most fundamental approach for Chinese; it is extremely fast, but it relies heavily on the dictionary, and the results are rather poor.

The first time it loads the dictionary, the approximate output is as follows:

Bash
$ git clone --recurse-submodules https://github.com/yuikns/argcv-analyzer-cn.git
$ cd argcv-analyzer-cn
$ mkdir build
$ cd build
$ cmake ..
$ make -j8
$ ./anzcn-simple load
loading lexicon from ../data/lexicon/webdict_with_freq.txt ...  done, size: [220725]
【/up 废土/uw 科学/uw 指南/uw ：/up f/aw 代表/uw 辐射/uw ，/up a/aw 代表/uw 原子/uw 】/up 
战争/uw ，/up 战争/uw 永不/uw 改变/uw 。/up 今天/uw 是/uw 《/up 辐射/uw 4/aw 》/up 上市/uw 
的/uw 日子/uw ，/up 虽然/uw 距离/uw 1/aw 代/uw 已经/uw 过去/uw 了/uw 十八/uw 年/uw ，/up 
但/uw 和/uw 战争/uw 一样/uw 不变/uw 的/uw ，/up 只有/uw 废土/uw ，/up 那个/uw 承载/uw 了/uw 
五十/uw 年代/uw 原子/uw 未来/uw 想象/uw 的/uw 废土/uw 。/up 现实/uw 的/uw 物理学/uw 虽然/uw 
和/uw 游戏/uw 中/uw 不同/uw ，/up 但/uw 我们/uw 都/uw 有/uw 同样/uw 的/uw 梦想/uw 和/uw 
同样/uw 的/uw 恐惧/uw .../ap some/aw english/aw word/aw 北京城/uw 连续/uw 几天/uw 阴天/uw 
，/up 然而/uw 在/uw 低空/uw 云层/uw 之上/uw 别有/uw 一番/uw 天地/uw 。/up 云海/uw 遮住/uw 
了/uw 城市/uw 灯光/uw ，/up 冬季/uw 夜空/uw 格外/uw 耀眼/uw 。/up 结婚/uw 的/uw 和/uw 尚未/uw 
结婚/uw 的/uw 同志/uw 们/uw 只/uw 听/uw 得/uw 一个/uw 女子/uw 低/uw 低/uw 应/uw 了/uw 一声/uw 
...