FileReadLine outputs gibberish

[virhonestum]

Hey,

I've coma accross a very odd problem. I want to download a CSV-File, and process the contents.

This is the extremely simplyfied AutoIT-Code, given the file is already downloaded:

$f= @ScriptDir & "\TestFile.csv"

$file = FileOpen($f,0)
Local $line = FileReadLine($file)
MsgBox(0,"",$line)

The downloaded CSV file I want to process contains something like this:

Artikelnummer;EAN-Code;Artikelname;Artikelgewicht;Beschreibung;Kurzbeschreibung;Eigenschaften;Technische-Daten;Bild1;Bild2;Bild3;Bild4;Bild5;Bild6;Bild7;Bild8;Lieferbar;"Lieferbar Ab";Versandzeit;"UVP-Preis inkl. MwSt.";"Preis1";"Preis2";Hersteller
L7335272;5420025602129; Mini Light XLR ;0.1000; JB Systems Schwanenhalsleuchte mit XLR Anschluss. ;;;;http://www.example.com/media/images/org/pic20070114153500a.jpg;;;;;;;;JA;; 1-3 Tage ;12,90;12,90;6,57; JB Systems 
L3320502;540207025601636; Mini Light LED BNC ;0.1000; JB Systems LED Schwanenhalsleuchte mit BNC Anschluss. ;;;;http://www.example.com/media/images/org/pic20061231171705a.jpg;;;;;;;;JA;; 1-3 Tage ;29,90;25,89;15,26; JB Systems 
L1332254;542002556023143; Mini Light LED XLR ;0.1000; JB Systems LED Schwanenhalsleuchte mit XLR Anschluss. ;;;;http://www.example.com/media/images/org/pic20061231171728a.jpg;;;;;;;;JA;; 1-3 Tage ;29,90;25,89;15,26; JB Systems 
L8302591;504200256280277; Spiegelkugel 10cm ;0.5000; JB Systems Spiegelkugel 10cm Durchmesser mit einer hohen Dichte durch 10 x 10 mm Echtglasspiegel. ;;;;http://www.example.com/media/images/org/pic20060324214825a.jpg;;;;;;;;JA;; 1-3 Tage ;5,50;4,90;2,81; JB Systems 
L7302932;542000256510222; Spiegelkugel 20cm ;0.8400; JB Systems Spiegelkugel 20cm Durchmesser mit einer hohen Dichte durch 10 x 10 mm Echtglasspiegel. ;;;;http://www.example.com/media/images/org/pic20060324214907a.jpg;;;;;;;;JA;; 1-3 Tage ;12,90;11,50;6,58; JB Systems 
L2350293;534200562064239; Spiegelkugel 30cm ;2.1300; JB Systems Spiegelkugel 30 cm Durchmesser mit einer hohen Dichte durch 10 x 10 mm Echtglasspiegel. ;;;;http://www.example.com/media/images/org/pic20060324214956a.jpg;;;;;;;;JA;; 1-3 Tage ;26,90;23,00;13,72; JB Systems 
L3302984;545200252024246; Spiegelkugel 40cm ;3.5000; JB Systems Spiegelkugel 40cm Durchmesser mit Sicherungsring und einer hohen Dichte durch 10 x 10 mm Echtglasspiegel. ;;;;http://www.example.com/media/images/org/pic20060324215050a.jpg;;;;;;;;JA;; 1-3 Tage ;54,90;49,00;28,00; JB Systems 
L9302495;542205056225600; Spiegelkugel 50cm ;5.3900; JB Systems Spiegelkugel 50cm Durchmesser mit Sicherungsring und einer hohen Dichte durch 10 x 10 mm Echtglasspiegel. ;;;;http://www.example.com/media/images/org/pic20060324215122a.jpg;;;;;;;;JA;; 1-3 Tage ;89,00;79,00;45,39; JB Systems

But the message box that pops up after FileReadLine contains this:

䅲瑩步汮畭浥爻䕁中䍯摥㭁牴楫敬湡浥㭁牴楫敬来睩捨琻䉥獣桲敩扵湧㭋畲穢敳捨牥楢畮朻䕩来湳捨慦瑥渻呥捨湩獣桥ⵄ慴敮㭂楬搱㭂楬搲㭂楬搳㭂楬搴㭂楬搵㭂楬搶㭂楬搷㭂楬搸㭌楥晥牢慲㬢䱩敦敲扡爠䅢∻噥牳慮摺敩琻≕噐ⵐ牥楳⁩湫氮⁍睓琮∻≐牥楳ㄢ㬢偲敩猲∻䡥牳瑥汬敲ੌ㜳㌵㈷㈻㔴㈰〲㔶〲ㄲ㤻M楮椠䱩杨琠塌刀㬰⸱〰〻J䈠卹獴敭猠卣桷慮敮桡汳汥畣桴攠浩琠塌删䅮獣桬畳献;㬻㭨瑴瀺⼯睷眮數慭灬攮捯洯浥摩愯業慧敳⽯牧⽰楣㈰〷〱ㄴㄵ㌵〰愮橰朻㬻㬻㬻㭊䄻㬀ㄭ㌠呡来;ㄲⰹ〻ㄲⰹ〻㘬㔷㬀䩂⁓祳瑥浳

I've attached both files I use. 

My guess is, that there's something wrong with the encoding, but I'm not sure how to fix it.

 

Thank you very much for your help

- virhonestum

Encodingtester.au3

TestFile.csv

[Jos]

There are NUL characters in your file so the file is opened with the wrong encoding.

Jos

[virhonestum]

1 hour ago, Jos said:

There are NUL characters in your file so the file is opened with the wrong encoding.

Jos

And how do I get rid of the NUL characters?Both _ReplaceStringInFile ($f, " ", "")  and _ReplaceStringInFile ($f, Chr(0), "") do not seem to work.

 

Thank you

- virhonestum

Edited January 29, 2017 by virhonestum

[TheDcoder]

Global Const $NULL = Chr(0)

[Subz]

Or just just use:

$file = FileOpen($f, 128) ;~ Opens the file using UTF8 encoding

[virhonestum]

15 minutes ago, TheDcoder said:

Global Const $NULL = Chr(0)

Thanks, but how exactly do I implement it? Just adding this line of code does nothing, and adding it and using $NULL in the _ReplaceInFile() call doesn't work either, since it is essentially the same as I already tried.

[virhonestum]

42 minutes ago, Subz said:

Or just just use:

$file = FileOpen($f, 128) ;~ Opens the file using UTF8 encoding

Thank You!!!! This worked!

[Jos]

Not really as that will still limit the string to the first NUL character in the file.

Jos

[Subz]

Sorry never checked the full output, I know the following works:

#include <Array.au3>
#include <Excel.au3>

Local $oExcel = _Excel_Open()
    If @error Then Exit

Local $sWorkbook = @ScriptDir & '\TestFile.csv'
Local $oWorkbook = _Excel_BookOpen($oExcel, $sWorkbook)
    If @error Then Exit

Local $aResult = _Excel_RangeRead($oWorkbook, Default, $oWorkbook.ActiveSheet.Usedrange)
    If @error Then Exit

_ArrayDisplay($aResult)