Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Welcome To Ask or Share your Answers For Others

Categories

0 votes
1.1k views
in Technique[技术] by (71.8m points)

string - Java unicode byte parsing

I'm just in the process of reading some data from a file as a stream of bytes, and I've just encountered some unicode strings that I'm not sure how best to handle.

Each character is using two bytes, with only the first seeming to contain actual data, so for example the string 'trust' is stored in the file as:

0x74 0x00(t) 0x72 0x00(r) ...and so on

Normally I'd just use a regex to replace the zeros with nothing and therefore remove the whitespace. However, the spaces between words within the file are implemented using 0x00 0x00, so trying to do a simple String 'replaceAll' is kind of messing it up a little.

I've tried playing around with the String encoding sets, such as 'ISO-8859-1' and 'UTF-8/16', but everytime I end up with white space.

I did create a simple regex to remove the double zero hex values, which is:

new String(bytes).replaceAll("[\00]{2,},"");

But this obviously only works for the double zero, and I'd really like to replace single zeros with nothing, and double zeros with a an actual ASCII/Unicode space character.

I could have sworn that one of the Java string format settings dealt with this kind of thing, but I might be wrong. So should I work on creating a regex to strip out the zeros, or does Java actually provide the mechanisms for doing it?

Thanks

See Question&Answers more detail:os

与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
Welcome To Ask or Share your Answers For Others

1 Reply

0 votes
by (71.8m points)

That's "UTF-16LE". 0x00 0x00 actually encodes the NUL character in UTF-16 so that's what you will get.

This encoding can encode about a million different characters, using 2 or 4 bytes per character. The first 256 characters are encoded with the second byte 0x00 and if the text only contains those it could be seen as useless, but it's required for the rest of the characters. For instance, the euro currency symbol would show up as 0xAC 0x20.


与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…
OGeek|极客中国-欢迎来到极客的世界,一个免费开放的程序员编程交流平台!开放,进步,分享!让技术改变生活,让极客改变未来! Welcome to OGeek Q&A Community for programmer and developer-Open, Learning and Share
Click Here to Ask a Question

...