1 post in this topic
I need to read log files into an array to search for errors. However when I display the array I get garbage or "chinese characters". Our developers say they are using UTF-8, but FileGetEncoding says the logs are "2048" or $FO_UTF16_BE_NOBOM (2048) = Use Unicode UTF16 Big Endian (without BOM) from the Encoding codes in FileOpen().
There is an app called Detenc that detects the encoding used by files. You have to guess, but it returns correctly when I set the Encoder for UTF-8. I understand Encoding is not etched in stone, but the first character of the file is a capital B, using HxD Hex Editor.
I even have another topic here about running PowerShell to reencode the file so AutoIt will store the file properly in the array - See:
So I am trying to figure out why AutoIt thinks my logs are not UTF-8.
Here is sample code:
#include <array.au3> #include <File.au3> Local $aRetArrayFile _FileReadToArray("C:\Logs\Myplayer1.log", $aRetArrayFile) _ArrayDisplay($aRetArrayFile) I won't post the results as it is illegible, but I did attach a screenshot of the _ArrayDisplay results, and this is the first line of the Log file:
BANNER 10/10/2017 15:56:00 ====================================================================== And the Hex from the beginning of the file:
42 41 4E 4E 45 52 20 31 30 2F 31 30 2F 32 30 31 37 20 31 34 3A 33 31 3A 33 35 20 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 3D 0D 0A 42 41 4E 4E 45 52 20 So I don't understand why AutoIt thinks the file is UTF16 BE.
If I can get the Powershell script running, I have a workaround.
BTW none of my other arrays display as garbage, just the log files.
Rereading my post, what seems to be missing is the question. I guess my question is, does anyone know why these logs are being displayed incorrectly?
I need help with unicode char Ã¼ I get some text from online json but if try to read 4 example Zürich I heave ZÃ¼rich.
How can I convert with autoit unicode to a clear character readable? thx
It's now possible to see the best compression ratio using LZMA, LZNT and Base64 compressions with differents combinations.
Nothing too complicate, you drag'n drop a file on the picture and script Test all compression types and return the ratios.
( Test duration depends of file size, slowest compression is LZNT, but all decompressions are fast )
Free to you after, to choose the compression(s) you want...
Yes, LZMA needs a dll ( embedded & compressed in script ) but brings a powerfull compression.
It opens scite with your file compressed to an au3 script with or without decompression function as you want.
Hold Left Shift key when clicking button for just copy script to clipboard.
Use the 3 compressions at a time works but doesn't give a good ratio, that's why i don't display it.
Usefull for little files you want include in your scripts !
No externals files needed, they are already in script.
Previous downloads : 1103
Source and Executable
BinaryToAu3Kompressor will be added to the next version of >SciTEHopper
Thanks to Ward for his >Base64.au3 and LZMA.au3, and trancexx for his >LZNT functions and his >Base64Decode function.
i need to save files with ANSI-Encoding. Since 22.214.171.124 Auto-It it doesn't work in any direction.
I tried the following:
#include <FileConstants.au3> FileDelete(@ScriptDir&"\Test.txt") $o = FileOpen(@ScriptDir&"\Test.txt", BitOR($FO_BINARY,$FO_ANSI,$FO_OVERWRITE)) FileWrite($o, "Test") FileClose($o) Or
#include <FileConstants.au3> FileDelete(@ScriptDir&"\Test.txt") $o = FileOpen(@ScriptDir&"\Test.txt", 514) FileWrite($o, "Test") FileClose($o) Both create UTF-8 encoded files.
What am i doing wrong?
This is a small UDF I put together just for fun that will embed and extract complete directory's in and from your script. It only has two functions, _EmbeddedDirectory_Create() and _EmbeddedDirectory_Extract(). The udf uses Wards machine code functions so they have pretty good speed. The example I set up packs all files located in Autoit Include directory. Both compression and decompression of the directory was less than 500ms on my pc which I was happy with. Let me if you have any problems or questions. Thanks
#include "EmbedDir.au3" ;Check location of autoit includes Global $sIncludeDir = 'C:\Program Files\AutoIt3\Include\' If Not FileExists($sIncludeDir) Then $sIncludeDir = 'C:\Program Files (x86)\AutoIt3\Include\' If Not FileExists($sIncludeDir) Then Exit (MsgBox(0, 'Directory Not Found', 'Unable to locate Includes Directory')) ;Create embedded directory function Global $sCompressed_Includes = _EmbeddedDirectory_Create($sIncludeDir, '_Extract_Includes') Global $iTime = (@extended / 1000) & ' ms' ;create example file and write extract function to it Global $hExtractExample = FileOpen(@ScriptDir & 'Extract_Example.au3', 2) FileWrite($hExtractExample, '#include "EmbedDir.au3"' & @LF & @LF & _ '$sExt = _Extract_Includes()' & @LF & _ '_EmbeddedDirectory_Extract($sExt, @ScriptDir & "AutoIt_Extracted")' & @LF & _ 'ConsoleWrite("Extract Time = " & @extended/1000 & "ms" & @LF)' & @LF & @LF & _ $sCompressed_Includes) FileClose($hExtractExample) ;Show stats of the directory we just embeded $aSize = DirGetSize($sIncludeDir,1) If IsArray($aSize) Then Msgbox(0,"Directory Details", _ "Size = " & Int($aSize/1024) & 'KB' & @LF & _ "Files = " & $aSize & @LF & _ "Dirs = " & $aSize & @LF & _ "Compression time = " & $iTime & @LF & @LF & _ "Open Extract_Example.au3 to extract") EndIf