UTF-8 Encoding Format

UTF-8 is an encoding format. More precisely, it is a container for storing Unicode. Unicode covers almost every character we need; to encompass all of that, a Unicode code point itself is a long integer. But when we actually store it, we need to split that Unicode value into several bytes. How do we serialize Unicode? UTF-8 is one solution.

The UTF-8 standard is documented in RFC3629. In short, we strip the leading zeros off the Unicode value, producing variable-length Unicode data, then write sequence information into the high bits of each byte. Because characters like ABCD and the rest of ASCII are used far more often, while many larger Unicode values are rarely used, and more importantly, when we only use data within the ASCII range UTF-8 is compatible with ASCII — simply wonderful.

RFC3629 specifies that each byte is stored as follows:

For a single-byte character, the highest bit is 0, and the remaining 7 bits hold the content. For characters longer than one byte, the leading byte is padded with as many 1s as there are total bytes, followed by 10 (in other words, the header uses byte count + 2 bits). From the second byte onward, the top two bits are 10. Because of this, UTF-8 can be at most 6 bytes long. (Any longer and there would be no bits left.)

Once you know the rule, splitting a UTF-8 string into characters is straightforward. First check the byte length, then advance that many bytes from the current position. I wrote a demo at here; feel free to take a look.

Calling it is also simple:

#include <string>
#include <vector>

#include "argcv/cxx/str/str_helper.h"

using std::string;
using std::vector;

using namespace argcv;

int main(int argc, char* argv[]) {
  vector<string> elems = Utf8Split("abcd\u00A0你好世界123\n");
  for (size_t i = 0; i < elems.size(); i++) {
    printf("%zu (%s):%lu\n", i,
      elems[i].c_str(), elems[i].length());
  }
  return 0;
}

The output is as follows:

$ ./run_example
0 (a):1
1 (b):1
2 (c):1
3 (d):1
4 ( ):2
5 (你):3
6 (好):3
7 (世):3
8 (界):3
9 (1):1
10 (2):1
11 (3):1
12 (
):1