Jump to content
kyo

Autoit3Wrapper character encoding problem

Recommended Posts

how can i tell scite to open the files in utf8 mode? As you can see in the pictures scite opens them with broken characters until i switch to utf8

When i create a new file (reight clich->New-Au3 file) it is created as DOS Ascii file.

When i use Compile with options it is always converted to an utf-8 file

What is the easiest way to make sure all files have the same format (how to convert) and what format is recommended?

Edited by Allow2010

Share this post


Link to post
Share on other sites

It does that automatically in case there is no BOM. It will check the file for Special charters and will open in UTF8 wo BOM when none are detected.
In case some special ansi characters are present in the file, like your test.au3, it will open in ANSI.

So when you take your posted example, both files are merged and then when opened it recognised the Special ansi character and will open as ANSI, which corrupts the UTF8 character from the main file.

Unsure how this need to be properly handled, but I am sure somebody can tell me. :)

Jos

Share this post


Link to post
Share on other sites

I just checked and the scripts in that file open in ANSI with mangled characters in scite, but correctly open as uft8 in other editors. It doesn't matter if you convert encoding to utf8 add a space and re-save - it opens as mangled again. The file will contain a non-ansi utf8 sequence so I would have thought it would be opened ok by scite. I'm sure it used to when I last tested that sort of thing.

That's just scite though, doesn't explain anything that happens in the tool chain.

 

Edited by Jon

Share this post


Link to post
Share on other sites

@Jon,Both files open fine when this settings are used ( not yet the default in the current production as I forgot to make that change in the last release)

utf8.auto.check=4

But that isn't really the issue here. We have 2 files of which one opens in UTF8 wo BOM and the other as ANSI. au3stripper simply concatenates the files without any conversion, so then when opened by SciTE it sees the special ANSI character and will obvuously corrupt the UTF character from the files file.

Jos

Edited by Jos

Share this post


Link to post
Share on other sites

Concatenating the files is going to be problematic. Better to just open the all files with auto-detect encoding, merge them in memory, and then always write out the result as utf8 wo BOM. Does au3stripper have auto-detect code in already? I can't recall if it's AutoIt program or a custom one.

Share this post


Link to post
Share on other sites

It is a custom program written in BCX ( a basic to C code generator) as I used that to convert the AutoIt3 script for gaining speed.
It doesn't have any UTF8 support so don't think it will be an simple change.

EDIT: Could always use the AutoIt3 script I already have for merging all code into one file first. :)

Jos 

Edited by Jos

Share this post


Link to post
Share on other sites

@Jon,Both files open fine when this settings are used ( not yet the default in the current production as I forgot to make that change in the last release)

utf8.auto.check=4

But that isn't really the issue here. We have 2 files of which one opens in UTF8 wo BOM and the other as ANSI. au3stripper simply concatenates the files without any conversion, so then when opened by SciTE it sees the special ANSI character and will obvuously corrupt the UTF character from the files file.

Jos

Sorry for this noob question, but how can i apply this setting to my SciTE ? It would at least help a bit in my everyday work...for now i can live without au3stripper...

Edited by Allow2010

Share this post


Link to post
Share on other sites

Just put that line in your SciTeUser.properties : SciTE Menu: Options/Open User Options File

As to using au3stripper: just ensure you only use utf8 with or without bom and you should be fine.

Jos

Share this post


Link to post
Share on other sites

Thank you.

I noticed something else, a bit offtopic:

i reinstalled my system and switched to win 10. I installed autoit, SciTE and the beta files as usual.

I now noticed that compiling my files creates temp files in the root of the drive where i compile.

They are named like this: ~CoXXXX and are 0 bytes. Never had this before on Win8.1.

Beta Bug or Win10 Problem?

Edited by Allow2010

Share this post


Link to post
Share on other sites

I now noticed that compiling my files creates temp files in the root of the drive where i compile.

They are named like this: ~CoXXXX and are 0 bytes.

Funny I have the same problem ~CoXXXX.tmp with 0 size.
Windows 7 Pro 64Bit

EDIT:

and I confirm that this is related to pressing F7 in S4A3

I wonder: Is it related to last change in AutoItWrapper or Au3Stripper ?

 

Edited by mLipok

Signature beginning:   Wondering who uses AutoIT and what it can be used for ?
* GHAPI UDF - modest begining - comunication with GitHub REST API Forum Rules *
ADO.au3 UDF     POP3.au3 UDF     XML.au3 UDF    How to use IE.au3  UDF with  AutoIt v3.3.14.x  for other useful stuff click the following button

Spoiler

Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind. 

My contribution (my own projects): * Debenu Quick PDF Library - UDF * Debenu PDF Viewer SDK - UDF * Acrobat Reader - ActiveX Viewer * UDF for PDFCreator v1.x.x * XZip - UDF * AppCompatFlags UDF * CrowdinAPI UDF * _WinMergeCompare2Files() * _JavaExceptionAdd() * _IsBeta() * Writing DPI Awareness App - workaround * _AutoIt_RequiredVersion() * Chilkatsoft.au3 UDF * TeamViewer.au3 UDF * JavaManagement UDF * VIES over SOAP * WinSCP UDF * GHAPI UDF - modest begining - comunication with GitHub REST APIErrorLog.au3 UDF - A logging Library *

My contribution to others projects or UDF based on  others projects: * _sql.au3 UDF  * POP3.au3 UDF *  RTF Printer - UDF * XML.au3 UDF * ADO.au3 UDF SMTP Mailer UDF * Dual Monitor resolution detection * * 2GUI on Dual Monitor System * _SciLexer.au3 UDF *

Useful links: * Forum Rules * Forum etiquette *  Forum Information and FAQs * How to post code on the forum * AutoIt Online Documentation * AutoIt Online Beta Documentation * SciTE4AutoIt3 getting started * Convert text blocks to AutoIt code * Games made in Autoit * Programming related sites * Polish AutoIt Tutorial * DllCall Code Generator * 

Wiki: Expand your knowledge - AutoIt Wiki * Collection of User Defined Functions * How to use HelpFile * Good coding practices in AutoIt * 

IE Related:  * How to use IE.au3  UDF with  AutoIt v3.3.14.x * Why isn't Autoit able to click a Javascript Dialog? * Clicking javascript button with no ID * IE document >> save as MHT file * IETab Switcher (by LarsJ ) * HTML Entities * _IEquerySelectorAll() (by uncommon) * IE in TaskScheduler

I encourage you to read: * Global Vars * Best Coding Practices * Please explain code used in Help file for several File functions * OOP-like approach in AutoIt * UDF-Spec Questions *  EXAMPLE: How To Catch ConsoleWrite() output to a file or to CMD *

"Homo sum; humani nil a me alienum puto" - Publius Terentius Afer
"Program are meant to be read by humans and only incidentally for computers and execute" - Donald Knuth, "The Art of Computer Programming"
:naughty:  :ranting:, be  :) and       \\//_.

Anticipating Errors :  "Any program that accepts data from a user must include code to validate that data before sending it to the data store. You cannot rely on the data store, ...., or even your programming language to notify you of problems. You must check every byte entered by your users, making sure that data is the correct type for its field and that required fields are not empty."

11

Signature last update: 2018-10-31

Share this post


Link to post
Share on other sites

What is the easiest way to make sure all files have the same format (how to convert) and what format is recommended?

;~ #AutoIt3Wrapper_Version=B
#include <File.au3>
#include <array.au3>



_Convert_Au3_Files_Encoding()

Func _Convert_Au3_Files_Encoding()

    ; beta include
    Local $sBetaFolder = RegRead("HKEY_LOCAL_MACHINE\SOFTWARE" & ((@OSArch = 'X64') ? "\Wow6432Node" : "") & "\AutoIt v3\AutoIt", "InstallDir") & '\Beta\Include\'
    If Not @error Then _Encoding_Converter($sBetaFolder)

    ; relase inculde
    _Encoding_Converter(RegRead("HKEY_LOCAL_MACHINE\SOFTWARE" & ((@OSArch = 'X64') ? "\Wow6432Node" : "") & "\AutoIt v3\AutoIt", "InstallDir") & '\Include\')

    _Encoding_Converter(@ScriptDir)
    _Encoding_Converter('..\MyProject1\')
    _Encoding_Converter('..\MyProject2\')
    _Encoding_Converter('..\MyProject3\')
    _Encoding_Converter('..\MyProject4\')
;~  _Encoding_Converter('')
;~  _Encoding_Converter('')

;~  ; and your own UDF folder
;~  Local $sFolderOwnUDF = FileSelectFolder('Choose your UDF folder','')
;~  if not @error then  _Encoding_Converter($sFolderOwnUDF)

EndFunc   ;==>_Convert_Au3_Files_Encoding


Func _Encoding_Converter($sFolder, $iENCODING = $FO_UTF8)
    ConsoleWrite($sFolder & @CRLF)
    Local $aIncludeList = _FileListToArray($sFolder, '*.au3', $FLTA_FILES, True)
    if @error then Return
    Local $sFileContent
    Local $hFile
    Local $iFile_Old_Encoding = -1
    Local $iFile_New_Encoding = -1
    Local $iFile_Attrib = ''
    Local $bFile_AttribChanged = False
    For $iFile = 1 To $aIncludeList[0]
        $iFile_Old_Encoding = FileGetEncoding($aIncludeList[$iFile])
        $iFile_Attrib = FileGetAttrib($aIncludeList[$iFile])
        If StringInStr($iFile_Attrib, 'R') Then
            FileSetAttrib($aIncludeList[$iFile], '-R')
            $bFile_AttribChanged = True
        Else
            $bFile_AttribChanged = False
        EndIf
        If $iFile_Old_Encoding = $iENCODING Then
            ConsoleWrite('> ')
        Else
            ConsoleWrite('! ')
            $sFileContent = FileRead($aIncludeList[$iFile])
;~          ClipPut($sFileContent)
;~          $sFileContent = ClipGet()

            FileDelete($aIncludeList[$iFile])
            $hFile = FileOpen($aIncludeList[$iFile], $FO_OVERWRITE + $iENCODING)
            ConsoleWrite(FileWrite($hFile, $sFileContent) & ' ')
            FileClose($hFile)
        EndIf
        If $bFile_AttribChanged Then FileSetAttrib($aIncludeList[$iFile], '+R')
        $iFile_New_Encoding = FileGetEncoding($aIncludeList[$iFile])
        ConsoleWrite('$iFile_[Old/New]_Encoding=[ ' & $iFile_Old_Encoding & ' / ' & $iFile_New_Encoding & ' ] ' & $aIncludeList[$iFile] & @CRLF)
    Next
EndFunc   ;==>_Encoding_Converter

 

Edited by mLipok
script cleanup

Signature beginning:   Wondering who uses AutoIT and what it can be used for ?
* GHAPI UDF - modest begining - comunication with GitHub REST API Forum Rules *
ADO.au3 UDF     POP3.au3 UDF     XML.au3 UDF    How to use IE.au3  UDF with  AutoIt v3.3.14.x  for other useful stuff click the following button

Spoiler

Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind. 

My contribution (my own projects): * Debenu Quick PDF Library - UDF * Debenu PDF Viewer SDK - UDF * Acrobat Reader - ActiveX Viewer * UDF for PDFCreator v1.x.x * XZip - UDF * AppCompatFlags UDF * CrowdinAPI UDF * _WinMergeCompare2Files() * _JavaExceptionAdd() * _IsBeta() * Writing DPI Awareness App - workaround * _AutoIt_RequiredVersion() * Chilkatsoft.au3 UDF * TeamViewer.au3 UDF * JavaManagement UDF * VIES over SOAP * WinSCP UDF * GHAPI UDF - modest begining - comunication with GitHub REST APIErrorLog.au3 UDF - A logging Library *

My contribution to others projects or UDF based on  others projects: * _sql.au3 UDF  * POP3.au3 UDF *  RTF Printer - UDF * XML.au3 UDF * ADO.au3 UDF SMTP Mailer UDF * Dual Monitor resolution detection * * 2GUI on Dual Monitor System * _SciLexer.au3 UDF *

Useful links: * Forum Rules * Forum etiquette *  Forum Information and FAQs * How to post code on the forum * AutoIt Online Documentation * AutoIt Online Beta Documentation * SciTE4AutoIt3 getting started * Convert text blocks to AutoIt code * Games made in Autoit * Programming related sites * Polish AutoIt Tutorial * DllCall Code Generator * 

Wiki: Expand your knowledge - AutoIt Wiki * Collection of User Defined Functions * How to use HelpFile * Good coding practices in AutoIt * 

IE Related:  * How to use IE.au3  UDF with  AutoIt v3.3.14.x * Why isn't Autoit able to click a Javascript Dialog? * Clicking javascript button with no ID * IE document >> save as MHT file * IETab Switcher (by LarsJ ) * HTML Entities * _IEquerySelectorAll() (by uncommon) * IE in TaskScheduler

I encourage you to read: * Global Vars * Best Coding Practices * Please explain code used in Help file for several File functions * OOP-like approach in AutoIt * UDF-Spec Questions *  EXAMPLE: How To Catch ConsoleWrite() output to a file or to CMD *

"Homo sum; humani nil a me alienum puto" - Publius Terentius Afer
"Program are meant to be read by humans and only incidentally for computers and execute" - Donald Knuth, "The Art of Computer Programming"
:naughty:  :ranting:, be  :) and       \\//_.

Anticipating Errors :  "Any program that accepts data from a user must include code to validate that data before sending it to the data store. You cannot rely on the data store, ...., or even your programming language to notify you of problems. You must check every byte entered by your users, making sure that data is the correct type for its field and that required fields are not empty."

11

Signature last update: 2018-10-31

Share this post


Link to post
Share on other sites

it happends on F5 in SciTE...

Which ran AutoIt3Wrapper and that in turn AU3Check & AutoIt3.exe or also Tidy?

Jos

Share this post


Link to post
Share on other sites

i tested with a script that has only one line

MsgBox(0,"","")

i just renamed tidy.exe and au3check.exe and press f5 and i get two temp files...

>"C:\Program Files (x86)\AutoIt3\SciTE\..\AutoIt3.exe" "C:\Program Files (x86)\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper.au3" /run /prod /ErrorStdOut /in "Z:\test.au3" /UserParams    
+>23:41:53 Starting AutoIt3Wrapper v.15.729.1555.6 SciTE v.3.6.0.0   Keyboard:00000407  OS:WIN_10/  CPU:X64 OS:X64    Environment(Language:0407)
+>         SciTEDir => C:\Program Files (x86)\AutoIt3\SciTE   UserDir => C:\Users\User\AppData\Local\AutoIt v3\SciTE\AutoIt3Wrapper   SCITE_USERHOME => C:\Users\User\AppData\Local\AutoIt v3\SciTE
! *** AU3CHECK Error: *** Skipping AU3Check: C:\Program Files (x86)\AutoIt3\au3check.exe Not Found !

>Running:(3.3.14.1):C:\Program Files (x86)\AutoIt3\autoit3.exe "Z:\test.au3"    
--> Press Ctrl+Alt+Break to Restart or Ctrl+Break to Stop
+>23:41:53 AutoIt3.exe ended.rc:0
+>23:41:53 AutoIt3Wrapper Finished.
>Exit code: 0    Time: 1.297

 

Edited by Allow2010

Share this post


Link to post
Share on other sites

I do test:

  1. rename Au3Check.exe to -Au3Check.exe
  2. press CTRL+F5 in S4A3

here is output from console:

>"C:\Program Files (x86)\AutoIt3\SciTE\..\AutoIt3.exe" "C:\Program Files (x86)\AutoIt3\SciTE\AutoIt3Wrapper\AutoIt3Wrapper.au3" /Prod /AU3check /in "Z:\test_ObjEvent.au3"
+>08:02:39 Starting AutoIt3Wrapper v.15.729.1555.6 SciTE v.3.6.0.0   Keyboard:00000415  OS:WIN_7/Service Pack 1  CPU:X64 OS:X64    Environment(Language:0415)
+>         SciTEDir => C:\Program Files (x86)\AutoIt3\SciTE   UserDir => C:\Users\user\AppData\Local\AutoIt v3\SciTE\AutoIt3Wrapper   SCITE_USERHOME => C:\Users\user\AppData\Local\AutoIt v3\SciTE 
! *** AU3CHECK Error: *** Skipping AU3Check: C:\Program Files (x86)\AutoIt3\au3check.exe Not Found !

+>08:02:39 AutoIt3Wrapper Finished.
>Exit code: 0    Time: 0.5599

How you see there was only AutoIt3Wrapper launched.
So I do a further test. I added In AutoIt3Wrapper.au3  some MsgBox for testing:

Func __TempFile($sDirectoryName = @TempDir, $sFilePrefix = "~", $sFileExtension = ".tmp", $iRandomLength = 7)
    ; his will create 2 files but this is to avoid same name when ran multiple times
    MsgBox(0, 'Test 1', 'Check you root directory for *.tmp files')
    Local $TempFile = _WinAPI_GetTempFileName($sDirectoryName, $sFilePrefix)
    If $sFileExtension <> ".tmp" Then
        FileCopy($TempFile, $TempFile & $sFileExtension, 1)
    Else
        $sFileExtension = ""
    EndIf
    MsgBox(0, 'Test 2 - Check Again', 'File: ' & $TempFile & $sFileExtension)
    Return $TempFile & $sFileExtension
EndFunc   ;==>__TempFile

it seams that the problem starts here:

Global $TempConsOut = __TempFile("", "~ConOut")

 

Edited by mLipok

Signature beginning:   Wondering who uses AutoIT and what it can be used for ?
* GHAPI UDF - modest begining - comunication with GitHub REST API Forum Rules *
ADO.au3 UDF     POP3.au3 UDF     XML.au3 UDF    How to use IE.au3  UDF with  AutoIt v3.3.14.x  for other useful stuff click the following button

Spoiler

Any of my own code posted anywhere on the forum is available for use by others without any restriction of any kind. 

My contribution (my own projects): * Debenu Quick PDF Library - UDF * Debenu PDF Viewer SDK - UDF * Acrobat Reader - ActiveX Viewer * UDF for PDFCreator v1.x.x * XZip - UDF * AppCompatFlags UDF * CrowdinAPI UDF * _WinMergeCompare2Files() * _JavaExceptionAdd() * _IsBeta() * Writing DPI Awareness App - workaround * _AutoIt_RequiredVersion() * Chilkatsoft.au3 UDF * TeamViewer.au3 UDF * JavaManagement UDF * VIES over SOAP * WinSCP UDF * GHAPI UDF - modest begining - comunication with GitHub REST APIErrorLog.au3 UDF - A logging Library *

My contribution to others projects or UDF based on  others projects: * _sql.au3 UDF  * POP3.au3 UDF *  RTF Printer - UDF * XML.au3 UDF * ADO.au3 UDF SMTP Mailer UDF * Dual Monitor resolution detection * * 2GUI on Dual Monitor System * _SciLexer.au3 UDF *

Useful links: * Forum Rules * Forum etiquette *  Forum Information and FAQs * How to post code on the forum * AutoIt Online Documentation * AutoIt Online Beta Documentation * SciTE4AutoIt3 getting started * Convert text blocks to AutoIt code * Games made in Autoit * Programming related sites * Polish AutoIt Tutorial * DllCall Code Generator * 

Wiki: Expand your knowledge - AutoIt Wiki * Collection of User Defined Functions * How to use HelpFile * Good coding practices in AutoIt * 

IE Related:  * How to use IE.au3  UDF with  AutoIt v3.3.14.x * Why isn't Autoit able to click a Javascript Dialog? * Clicking javascript button with no ID * IE document >> save as MHT file * IETab Switcher (by LarsJ ) * HTML Entities * _IEquerySelectorAll() (by uncommon) * IE in TaskScheduler

I encourage you to read: * Global Vars * Best Coding Practices * Please explain code used in Help file for several File functions * OOP-like approach in AutoIt * UDF-Spec Questions *  EXAMPLE: How To Catch ConsoleWrite() output to a file or to CMD *

"Homo sum; humani nil a me alienum puto" - Publius Terentius Afer
"Program are meant to be read by humans and only incidentally for computers and execute" - Donald Knuth, "The Art of Computer Programming"
:naughty:  :ranting:, be  :) and       \\//_.

Anticipating Errors :  "Any program that accepts data from a user must include code to validate that data before sending it to the data store. You cannot rely on the data store, ...., or even your programming language to notify you of problems. You must check every byte entered by your users, making sure that data is the correct type for its field and that required fields are not empty."

11

Signature last update: 2018-10-31

Share this post


Link to post
Share on other sites

Currently on the road, but will have some time in the next couple of days to have a look at this in detail.

Jos

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

×
×
  • Create New...