Skip to content
Advertisement

Extract substring by utf-8 byte positions

I have a string and start and length with which to extract a substring. Both positions (start and length) are based on the byte offsets in the original UTF8 string.

However, there is a problem:

The start and length are in bytes, so I cannot use “substring”. The UTF8 string contains several multi-byte characters. Is there a hyper-efficient way of doing this? (I don’t need to decode the bytes…)

Example: var orig = ‘你好吗?’

The s,e might be 3,3 to extract the second character (好). I’m looking for

JavaScript

Help!

Update #1 In C/C++ I would just cast it to a byte array, but not sure if there is an equivalent in javascript. BTW, yes we could parse it into a byte array and parse it back to a string, but it seems that there should be a quick way to cut it at the right place. Imagine that ‘orig’ is 1000000 characters, and s = 6 bytes and l = 3 bytes.

Update #2 Thanks to zerkms helpful re-direction, I ended up with the following, which does NOT work right – works right for multibyte but messed up for single byte.

JavaScript

Update #3 I don’t think shifting the char code really works. I’m reading two bytes when the correct answer is three… somehow I always forget this. The codepoint is the same for UTF8 and UTF16, but the number of bytes taken up on encoding depends on the encoding!!! So this is not the right way to do this.

Advertisement

Answer

I had a fun time fiddling with this. Hope this helps.

Because Javascript does not allow direct byte access on a string, the only way to find the start position is a forward scan.


Update #3 I don’t think shifting the char code really works. I’m reading two bytes when the correct answer is three… somehow I always forget this. The codepoint is the same for UTF8 and UTF16, but the number of bytes taken up on encoding depends on the encoding!!! So this is not the right way to do this.

This is not correct – Actually there is no UTF-8 string in javascript. According to the ECMAScript 262 specification all strings – regardless of the input encoding – must be internally stored as UTF-16 (“[sequence of] 16-bit unsigned integers”).

Considering this, the 8 bit shift is correct (but unnecessary).

Wrong is the assumption that your character is stored as a 3-byte sequence…
In fact, all characters in a JS (ECMA-262) string are 16 bit (2 byte) long.

This can be worked around by converting the multibyte characters to utf-8 manually, as shown in the code below.


UPDATE This solution doesn’t handle codepoints >= U+10000 including emoji. See APerson’s Answer for a more complete solution.


See the details explained in my example code:

JavaScript
User contributions licensed under: CC BY-SA
5 People found this is helpful
Advertisement