Jump to content

UTF-8 encoding ? !! \u2019 \u2013 \u2018


Recommended Posts

Hello

 

i have lots of text like this

 

With Viber, everyone in the world can connect. Freely. More than 516 million Viber users text, make HD-quality phone and video calls, and send photo and video messages worldwide over Wifi or 3G - for free.* Viber Out can be used to make calls to non-Viber mobile and landline numbers at low rates. Viber is available for many smartphones and platforms. Viber is compatible with and optimized for Android tablets! Use Viber on your tablet and phone simultaneously.\nOn Viber, your phone number is your ID. The app syncs with your mobile contact list, automatically detecting which of your contacts have Viber. \n\n\u2022 Message your friends\n\u2022 Make free phone and video calls with HD sound quality\n\u2022 Share photos, video messages, voice messages, locations, stickers and emoticons\n\u2022 Download stickers and animated stickers from the Sticker Market; sort and reorder your stickers.\n\u2022 Create group messages with up to 100 participants; become an Admin and manage your group chats \u2013 edit info and delete participants\n\u2022 Follow Public Chats \u2013 get on the inside with your favorite personalities; watch their conversations unfold in real-time, \u2018like\u2019 comments and share multi-media content\n\u2022 Play games with Viber characters, Violet and Legcat; see how many coins you can earn\n\u2022 Android Wear support \u2013 send and receive messages from your watch\n\u2022 Push notifications guarantee that you never miss a message or call, even when Viber is off\n\u2022 Integration with native contact list for calls and messages\n\u2022 Support for the Viber Desktop application on Windows and Mac\n\nLocalized to: Arabic, Catalan, Chinese (SP), Chinese (TR), Croatian, Czech, Danish, Dutch, Finnish, French, German, Greek, Hebrew, Hindi, Hungarian, Indonesian, Italian, Japanese, Korean, Malay, Norwegian, Polish, Portuguese (BR), Portuguese (PT), Romanian, Russian, Slovak, Spanish, Swedish, Tagalog, Thai, Turkish, Ukrainian and Vietnamese\nViber is completely free with no advertising. \nWe value your privacy. \n\nFollow us for updates and news:\nFacebook - http:\/\/facebook.com\/viber\nTwitter - http:\/\/twitter.com\/viber\n\n(*) Network data charges may apply\n\n\n\n\n\n\n\n\n\n

i have no idea what does this \u stuff means, after googling it i found that it is some sort of utf encoding

 

how could i write this text to a file without loosing any characters and getting rid of \u  ?

 

 

 

EDIT: after more digging i found that i need "Converting Unicode Entities to Unicode Text" any ideas hoe to do that with autoit ?

Edited by Alexxander

Link to comment
Share on other sites

I am not sure that you can convert this text because it is already formatted as ansi. You could use StringRegExpReplace to replace the unwanted characters. I made a simple one that gets rid of most of the annoying ones, not just the \u. The $str is the string that you posted.

$str = "With Viber, everyone in the world can connect. Freely. More than 516 million Viber users text, make HD-quality phone and video calls, and send photo and video messages worldwide over Wifi or 3G - for free.* Viber Out can be used to make calls to non-Viber mobile and landline numbers at low rates. Viber is available for many smartphones and platforms. Viber is compatible with and optimized for Android tablets! Use Viber on your tablet and phone simultaneously.\nOn Viber, your phone number is your ID. The app syncs with your mobile contact list, automatically detecting which of your contacts have Viber. \n\n\u2022 Message your friends\n\u2022 Make free phone and video calls with HD sound quality\n\u2022 Share photos, video messages, voice messages, locations, stickers and emoticons\n\u2022 Download stickers and animated stickers from the Sticker Market; sort and reorder your stickers.\n\u2022 Create group messages with up to 100 participants; become an Admin and manage your group chats \u2013 edit info and delete participants\n\u2022 Follow Public Chats \u2013 get on the inside with your favorite personalities; watch their conversations unfold in real-time, \u2018like\u2019 comments and share multi-media content\n\u2022 Play games with Viber characters, Violet and Legcat; see how many coins you can earn\n\u2022 Android Wear support \u2013 send and receive messages from your watch\n\u2022 Push notifications guarantee that you never miss a message or call, even when Viber is off\n\u2022 Integration with native contact list for calls and messages\n\u2022 Support for the Viber Desktop application on Windows and Mac\n\nLocalized to: Arabic, Catalan, Chinese (SP), Chinese (TR), Croatian, Czech, Danish, Dutch, Finnish, French, German, Greek, Hebrew, Hindi, Hungarian, Indonesian, Italian, Japanese, Korean, Malay, Norwegian, Polish, Portuguese (BR), Portuguese (PT), Romanian, Russian, Slovak, Spanish, Swedish, Tagalog, Thai, Turkish, Ukrainian and Vietnamese\nViber is completely free with no advertising. \nWe value your privacy. \n\nFollow us for updates and news:\nFacebook - http:\/\/facebook.com\/viber\nTwitter - http:\/\/twitter.com\/viber\n\n(*) Network data charges may apply\n\n\n\n\n\n\n\n\n\n"

$str = StringRegExpReplace($str, "(\\n\\u2022|\\u2013|\\u2019|\\u2018|\\n|\\)", "")

MsgBox(0, "", $str)

 

Link to comment
Share on other sites

$string = "With Viber, everyone in the world can connect. Freely. More than 516 million Viber users text, make HD-quality phone and video calls, and send photo and video messages worldwide over Wifi or 3G - for free.* Viber Out can be used to make calls to non-Viber mobile and landline numbers at low rates. Viber is available for many smartphones and platforms. Viber is compatible with and optimized for Android tablets! Use Viber on your tablet and phone simultaneously.\nOn Viber, your phone number is your ID. The app syncs with your mobile contact list, automatically detecting which of your contacts have Viber. \n\n\u2022 Message your friends\n\u2022 Make free phone and video calls with HD sound quality\n\u2022 Share photos, video messages, voice messages, locations, stickers and emoticons\n\u2022 Download stickers and animated stickers from the Sticker Market; sort and reorder your stickers.\n\u2022 Create group messages with up to 100 participants; become an Admin and manage your group chats \u2013 edit info and delete participants\n\u2022 Follow Public Chats \u2013 get on the inside with your favorite personalities; watch their conversations unfold in real-time, \u2018like\u2019 comments and share multi-media content\n\u2022 Play games with Viber characters, Violet and Legcat; see how many coins you can earn\n\u2022 Android Wear support \u2013 send and receive messages from your watch\n\u2022 Push notifications guarantee that you never miss a message or call, even when Viber is off\n\u2022 Integration with native contact list for calls and messages\n\u2022 Support for the Viber Desktop application on Windows and Mac\n\nLocalized to: Arabic, Catalan, Chinese (SP), Chinese (TR), Croatian, Czech, Danish, Dutch, Finnish, French, German, Greek, Hebrew, Hindi, Hungarian, Indonesian, Italian, Japanese, Korean, Malay, Norwegian, Polish, Portuguese (BR), Portuguese (PT), Romanian, Russian, Slovak, Spanish, Swedish, Tagalog, Thai, Turkish, Ukrainian and Vietnamese\nViber is completely free with no advertising. \nWe value your privacy. \n\nFollow us for updates and news:\nFacebook - http:\/\/facebook.com\/viber\nTwitter - http:\/\/twitter.com\/viber\n\n(*) Network data charges may apply\n\n\n\n\n\n\n\n\n\n"
$string = StringReplace($string, '\n', @CRLF)
$string = StringReplace($string, '\/', '/')
$string = Execute('"' & StringRegExpReplace($string, '\\u(\d\d\d\d)', '" & Chr($1) & "') & '"')

 

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Link to comment
Share on other sites

$string = "With Viber, everyone in the world can connect. Freely. More than 516 million Viber users text, make HD-quality phone and video calls, and send photo and video messages worldwide over Wifi or 3G - for free.* Viber Out can be used to make calls to non-Viber mobile and landline numbers at low rates. Viber is available for many smartphones and platforms. Viber is compatible with and optimized for Android tablets! Use Viber on your tablet and phone simultaneously.\nOn Viber, your phone number is your ID. The app syncs with your mobile contact list, automatically detecting which of your contacts have Viber. \n\n\u2022 Message your friends\n\u2022 Make free phone and video calls with HD sound quality\n\u2022 Share photos, video messages, voice messages, locations, stickers and emoticons\n\u2022 Download stickers and animated stickers from the Sticker Market; sort and reorder your stickers.\n\u2022 Create group messages with up to 100 participants; become an Admin and manage your group chats \u2013 edit info and delete participants\n\u2022 Follow Public Chats \u2013 get on the inside with your favorite personalities; watch their conversations unfold in real-time, \u2018like\u2019 comments and share multi-media content\n\u2022 Play games with Viber characters, Violet and Legcat; see how many coins you can earn\n\u2022 Android Wear support \u2013 send and receive messages from your watch\n\u2022 Push notifications guarantee that you never miss a message or call, even when Viber is off\n\u2022 Integration with native contact list for calls and messages\n\u2022 Support for the Viber Desktop application on Windows and Mac\n\nLocalized to: Arabic, Catalan, Chinese (SP), Chinese (TR), Croatian, Czech, Danish, Dutch, Finnish, French, German, Greek, Hebrew, Hindi, Hungarian, Indonesian, Italian, Japanese, Korean, Malay, Norwegian, Polish, Portuguese (BR), Portuguese (PT), Romanian, Russian, Slovak, Spanish, Swedish, Tagalog, Thai, Turkish, Ukrainian and Vietnamese\nViber is completely free with no advertising. \nWe value your privacy. \n\nFollow us for updates and news:\nFacebook - http:\/\/facebook.com\/viber\nTwitter - http:\/\/twitter.com\/viber\n\n(*) Network data charges may apply\n\n\n\n\n\n\n\n\n\n"
$string = StringReplace($string, '\n', @CRLF)
$string = StringReplace($string, '\/', '/')
$string = Execute('"' & StringRegExpReplace($string, '\\u(\d\d\d\d)', '" & Chr($1) & "') & '"')

 

​thank you for your simple solution ...

but this way i loosed the symbols 

ex: 

\u2022

 

this is a bullet symbol ""

 

any other ideas ?

Link to comment
Share on other sites

Oh sorry, it was a typo. Chr($1) make that ChrW(0x$1)

Edited by jchd

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Link to comment
Share on other sites

Bro i'am facing a problem  with this text using your edited code

 

The incredible new Dungeon Crawler taking the world by storm! Over 10 million downloads worldwide, with all new exciting content now available! Grow your character and restore the cosmic balance in an exciting free-to-play online adventure! AppleTell \u2013 named \u201cBest iOS Game of E3 2014\u201d\n\nTouchArcade \u2013 \u201cTaichi Panda has all the necessary components that make up a great hack 'n slash action game.\u201d\n\n148Apps \u2013 \u201cThe game looks like it has some pretty nice graphics and fast play style.\u201d\n\n==Features==\n\nA Stunning Fantasy Adventure\n\nEnter a world of goblins, bandits, and pandas as you adventure through graphically stunning maps in full 3D environments. Bash your way to the center of the mystery with a fast and fluid animation style built on full-body motion capture.\n\nHack \u2018n Slash Action\n\nFight through hordes of enemies with a fast-paced combat system. Use a mixture of speed, strength and skill, combining aerial attacks and combo moves to devastate the enemy. Explore twisting labyrinths, avoid precarious traps, and defeat powerful bosses!\n\nUpgrade Everything\n\nLevel up your character to unlock powerful new abilities. Upgrade and refine your gear to bolster your power, and unleash elemental magic through ancient runes. Develop your Pets, special battle companions with their own unique progression and skills.\n\nPlay in Real-time with Friends\n\nPlay in real-time multiplayer with friends and strangers. Become the ultimate Taichi champion with PvP arenas and battlefields. Team up with fellow adventurers in group dungeons to face off against formidable Bosses. Form Guilds with friends and compatriots, and expand your territory in Guild vs Guild war mode!\n\n==Connect==\n\nVisit our official site to learn more about Taichi Panda: http:\/\/panda.snail.com\/en \nJoin the Taichi Panda community and learn more on Facebook and our official Forum \nFacebook Page: https:\/\/www.facebook.com\/TaichiPandaEN \nCommunity Forum: http:\/\/pandabbs.snail.com\/\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n

Any ideas ? 

Link to comment
Share on other sites

Jeez, I apologize for not drinking a second coffee for the second eye!

$string = "The incredible new Dungeon Crawler taking the world by storm! Over 10 million downloads worldwide, with all new exciting content now available! Grow your character and restore the cosmic balance in an exciting free-to-play online adventure! AppleTell \u2013 named \u201cBest iOS Game of E3 2014\u201d\n\nTouchArcade \u2013 \u201cTaichi Panda has all the necessary components that make up a great hack 'n slash action game.\u201d\n\n148Apps \u2013 \u201cThe game looks like it has some pretty nice graphics and fast play style.\u201d\n\n==Features==\n\nA Stunning Fantasy Adventure\n\nEnter a world of goblins, bandits, and pandas as you adventure through graphically stunning maps in full 3D environments. Bash your way to the center of the mystery with a fast and fluid animation style built on full-body motion capture.\n\nHack \u2018n Slash Action\n\nFight through hordes of enemies with a fast-paced combat system. Use a mixture of speed, strength and skill, combining aerial attacks and combo moves to devastate the enemy. Explore twisting labyrinths, avoid precarious traps, and defeat powerful bosses!\n\nUpgrade Everything\n\nLevel up your character to unlock powerful new abilities. Upgrade and refine your gear to bolster your power, and unleash elemental magic through ancient runes. Develop your Pets, special battle companions with their own unique progression and skills.\n\nPlay in Real-time with Friends\n\nPlay in real-time multiplayer with friends and strangers. Become the ultimate Taichi champion with PvP arenas and battlefields. Team up with fellow adventurers in group dungeons to face off against formidable Bosses. Form Guilds with friends and compatriots, and expand your territory in Guild vs Guild war mode!\n\n==Connect==\n\nVisit our official site to learn more about Taichi Panda: http:\/\/panda.snail.com\/en \nJoin the Taichi Panda community and learn more on Facebook and our official Forum \nFacebook Page: https:\/\/www.facebook.com\/TaichiPandaEN \nCommunity Forum: http:\/\/pandabbs.snail.com\/\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n"
$string = StringReplace($string, '\n', @CRLF)
$string = StringReplace($string, '\/', '/')
$string = Execute('"' & StringRegExpReplace($string, '\\u([[:xdigit:]]{4})', '" & ChrW(0x$1) & "') & '"')

 

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Link to comment
Share on other sites

All you may now encounter are different escapes, like \r \t or similar. Now the \uxxxx are correctly changed into the corresponding Unicode character. Once again sorry for not giving my previous answers enough attention.

This wonderful site allows debugging and testing regular expressions (many flavors available). An absolute must have in your bookmarks.
Another excellent RegExp tutorial. Don't forget downloading your copy of up-to-date pcretest.exe and pcregrep.exe here
RegExp tutorial: enough to get started
PCRE v8.33 regexp documentation latest available release and currently implemented in AutoIt beta.

SQLitespeed is another feature-rich premier SQLite manager (includes import/export). Well worth a try.
SQLite Expert (freeware Personal Edition or payware Pro version) is a very useful SQLite database manager.
An excellent eBook covering almost every aspect of SQLite3: a must-read for anyone doing serious work.
SQL tutorial (covers "generic" SQL, but most of it applies to SQLite as well)
A work-in-progress SQLite3 tutorial. Don't miss other LxyzTHW pages!
SQLite official website with full documentation (may be newer than the SQLite library that comes standard with AutoIt)

Link to comment
Share on other sites

Of course just replacing the \u with &#x and ending the code point with ; will also result in the correct char being displaied in the html - try:

<html>
<head>
<title></title>
</head>
<body>
&#x2022;<br>
&#x201d;
</body>
</html>

 

Edited by Jury
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...