Reading/creating/updating/saving XML

[Jokerman]

First off, I finally created an account and I'd like to thank all of the wonderful people on this forum who have helped me immensely through the years without even knowing it. I've been coming here for over a decade, on and off, as I've found needs for automation in my various IT positions. I'd love to call everyone out but there are far too many! With that said, in no particular order, I'd like to extend special thanks to Melba23, Jos, guinness, RTFC, trancexx, mLipok, Eltorro, UEZ, Valuater and Ward. It seems like every time I run across a particularly insightful or informative thread I also find at least one or your names - either posting or referred to a previous post as the solution. You know your stuff and it's obvious you deeply care for the community and helping people. While you'll likely never hear it from nearly the number of people that you benefit from your advice, I just wanted to make sure you know there are a TON of appreciative people out there who've never expressed it directly.

So, my issue at this point is I'm having a hard time wrapping my head around the best way to deal with XML. While code is always highly descriptive, I'm mostly looking for a medium- to high-level overview of how you would tackle this problem using the XML UDF last updated by mLipok (1.1.1.13)...

Here's my XML file - data.xml:

expandcollapse popup

<?xml version="1.0" encoding="utf-8"?>
<ArrayOfData xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <Data>
    <Enabled>true</Enabled>
    <dataType>T1</dataType>
    <dataDetails xsi:type="Type1">
      <field1>name1</field1>
      <field2>email1</field2>
      <field3>choice1</field3>
      <field4>choice2</field4>
      <field5>choice3</field5>
      <field6>choice4</field6>
      <field7>choice5</field7>
      <field8>choice7</field8>
    </dataDetails>
    <optionSetting>true</optionSetting>
  </Data>
  <Data>
    <Enabled>true</Enabled>
    <dataType>T1</dataType>
    <dataDetails xsi:type="Type1">
      <field1>name2</field1>
      <field2>email2</field2>
      <field3>choice1</field3>
      <field4>choice2</field4>
      <field5>choice3</field5>
      <field6>choice4</field6>
      <field7>choice5</field7>
      <field8>choice7</field8>
    </dataDetails>
    <optionSetting>true</optionSetting>
  </Data>
  <Data>
    <Enabled>true</Enabled>
    <dataType>T2</dataType>
    <dataDetails xsi:type="Type2">
      <field1>name3</field1>
      <field2>email3</field2>
      <field6>choice6</field6>
      <field8>choice7</field8>
    </dataDetails>
    <optionSetting>true</optionSetting>
  </Data>
  <Data>
    <Enabled>true</Enabled>
    <dataType>T1</dataType>
    <dataDetails xsi:type="Type1">
      <field1>name4</field1>
      <field2>email4</field2>
      <field3>choice1</field3>
      <field4>choice2</field4>
      <field5>choice3</field5>
      <field6>choice4</field6>
      <field7>choice5</field7>
      <field8>choice7</field8>
    </dataDetails>
    <optionSetting>true</optionSetting>
  </Data>
</ArrayOfData>

I also have an AutoIt 2D array ($aNewData[$x][2] = [["name5"][$aName5],["name6"][$aName6],["name7"][$aName7]]) filled with 2D arrays containing data that needs to be entered.

I want to open the XML file (data.xml) as an object, check the T1>field1 values against the values in $aNewData[$x][0], populate any non-duplicates from ($aNewData[$x][1])[$y][$z] into the XML object, and save that XML object. Before modifying data.xml I need to make a backup - I have a backup function but I'm unsure when initial modification of the XML takes place so I don't know at what point I need to make a backup (I like to make it right before the point of modification so it only happens if/when the file is actually going to be changed). Also, if the file is blank or missing I need to create it from scratch and then populate with the values from ($aNewData[$x][1])[$y][$z].

Important information: I need nothing from the T2/Type2 nodes but they must remain in place if they exist. There are actually 30+ child nodes of dataDetails that are uniquely named and, while not named sequentially like my example shows, they are in a specific order that's dependent on "dataType" (T1/T2). "field1" is the only node value that cannot have duplicated values.

A second feature I need to include is the ability to import the data from data.xml and be able to easily manipulate it. I'm already dealing with several formats and the common denominator is AutoIt arrays - I currently import all data into arrays, manipulate as needed, and then format those arrays out to the required type (Send/ControlSend/Clipboard/JSON/etc). So, based on that and using my example data.xml, what would the best/most efficient way to read all of the "field#" key/value pairs into an AutoIt array?

As far as my experience with the XML UDF - it's very little. I've done some playing with the examples provided with the XML UDF, and I believe I have a basic understanding of how it work, but any tips/tricks/gotchas would be greatly appreciated.

To any who've made it this far - thank you for taking the time to read through my issue! Sorry, I know it's long but I wanted to try to avoid the mistakes made by so many in their first posts (incorrect or lack of information, trying to fix a symptom rather than a solution to reach the goal, etc). But if I failed that objective and you need further information, please don't hesitate to ask.

Thanks again! I'm looking forward to any assistance or information anyone can provide! 

 

data.xml

[jdelaney]

Do a forum search for microsoft.xmldom

[Subz]

Hopefully the following will help, I gather you can add in the checks against the array, you probably want to comment out the _Add_DataNode() function as it will keep adding a Data node section to the end of your sheet, it was basically just an example.

expandcollapse popup

#include <XML.au3>

;~ Create a blank template file
;~ ArrayData - Template.xml
Local $fXMLTemplate = @ScriptDir & "\Template.xml"
If FileExists($fXMLTemplate) = 0 Then
    Local $hXMLTemplate = FileOpen($fXMLTemplate, 10)
    $sXMLTemplate = '<?xml version="1.0" encoding="utf-8"?>' & @CRLF
    $sXMLTemplate &= '<ArrayOfData xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">' & @CRLF
    $sXMLTemplate &= '</ArrayOfData>' & @CRLF
    FileWrite($hXMLTemplate, $sXMLTemplate)
    FileClose($hXMLTemplate)
EndIf

Local $fXMLFileName = @ScriptDir & "\FileName.xml"
    If FileExists($fXMLFileName) = 0 Then FileCopy($fXMLTemplate, $fXMLFileName, 9)

Local $oRoot_Node, $oData_Node, $oData_SubNode
Local $oErrorHandler = ObjEvent("AutoIt.Error", "_XmlErrFunc")
Local $oXMLDoc = ObjCreate("MSXML2.DOMDocument")
    $oXMLDoc.validateOnParse = True
    $oXMLDoc.load($fXMLFileName)
Local $oDataNodes_Enum = $oXMLDoc.selectNodes("ArrayOfData/Data")

;~ Just as an example
;~ Add a new Data node
_Add_DataNode()

;~ Check existing nodes
If IsObj($oDataNodes_Enum) Then
    For $oData_Enum In $oDataNodes_Enum
        For $i = 0 To ($oData_Enum.childNodes.length - 1)
            If $oData_Enum.childNodes.item($i).tagName = "dataType" And $oData_Enum.childNodes.item($i).Text = "T2" Then ContinueLoop
            If $oData_Enum.childNodes.item($i).tagName = "dataDetails" Then
                For $j = 0 To ($oData_Enum.childNodes.item($i).childNodes.length - 1)
                    ConsoleWrite($oData_Enum.childNodes.item($i).childNodes.item($j).Text & @CRLF)
                Next
            EndIf
        Next
    Next
    ;~ Document is modified only after saving the document
    $sXMLDoc = _XML_TIDY($oXMLDoc)
    $sXMLDoc.save($fXMLFileName)
Else
    ;~ Add Data Node
    _Add_DataNode()
    ;~ Document is modified only after saving the document
;~  _XML_TIDY($oXMLDoc)
    $oXMLDoc.save($oXMLDoc)
EndIf

Func _Add_DataNode()
    $oRoot_Node = $oXMLDoc.documentElement
    $oData_Node = _XML_AddElement($oRoot_Node, "Data")
        _XML_AddElement($oData_Node, "Enabled", "true")
        _XML_AddElement($oData_Node, "dataType", "T1")
    $oData_SubNode = _XML_AddElement($oData_Node, "dataDetails")
        _XML_AddAttrib($oData_SubNode, "xsi:Type", "Type1")
        _XML_AddElement($oData_SubNode, "field1", "name1")
        _XML_AddElement($oData_SubNode, "field2", "email1")
        _XML_AddElement($oData_SubNode, "field3", "choice1")
        _XML_AddElement($oData_SubNode, "field4", "choice2")
        _XML_AddElement($oData_SubNode, "field5", "choice3")
        _XML_AddElement($oData_SubNode, "field6", "choice4")
        _XML_AddElement($oData_SubNode, "field7", "choice5")
        _XML_AddElement($oData_SubNode, "field8", "choice6")
    _XML_AddElement($oData_Node, "optionSetting", "true")
    
;~ Save XML with indentations
    ;~  $sXMLDoc = _XML_TIDY($oXMLDoc)
    ;~  Local $hFileOpen = FileOpen($fXMLFileName, 2)
    ;~      FileWrite($fXMLFileName, $sXMLDoc)
    ;~  FileClose($hFileOpen)

;~ Save XML without indentations
    $oXMLDoc.save($fXMLFileName)
EndFunc

Func _XML_AddElement($_oNode, $_sTagName = "", $_sText = "")
    Local $oNode = $oXMLDoc.createElement($_sTagName)
    If $_sText <> "" Then $oNode.Text = $_sText
    $_oNode.appendChild($oNode)
    Return $oNode
EndFunc

Func _XML_AddAttrib($_oAttrib, $_sAttribName = "", $_sNodeValue = "")
    Local $oAttrib = $oXMLDoc.createAttribute($_sAttribName)
    If $_sNodeValue <> "" Then $oAttrib.nodeValue = $_sNodeValue
    $_oAttrib.setAttributeNode($oAttrib)
    Return $oAttrib
EndFunc

 

[Earthshine]

Nm 

Edited April 10, 2018 by Earthshine

[Jokerman]

14 hours ago, jdelaney said:

Do a forum search for microsoft.xmldom

jdelaney,

Doh! I knew there would be names I'd forget and you, sir, are certainly one of them! Thank you for so much of the understanding I've gained in using AutoIt due, in no small part, to your excellent advice. I actually have another tab open with your Excel XML UDF that I plan on taking a look at when my current task is complete. I'm curious if the XML Excel files your UDF creates are in any way related or similar to the XML files held within an Excel XLSX file (they're simply compressed files containing XML files). If the UDF can't already, I think with some modification it could likely create standard XLSX files as quickly and easily as Excel XML files.

I've already familiarized myself with XPath, as well looked at the Microsoft XML DOM methods on MSDN (I have a tab open for looking up functionality as needed). I was looking more for a general idea of how best to work back and forth from XML to AutoIt array. For instance, if when going from XML to array it's easiest to simply loop through the nodes and rip XML key/value pairs to an array then I have no problem with that. I just know that right now my experience and knowledge when it comes to dealing with objects in general is seriously lacking and I thought there might be a more elegant or efficient way of obtaining the objective. It's fine if there isn't, I just don't know what I don't know.

 

1 hour ago, Subz said:

Hopefully the following will help, I gather you can add in the checks against the array, you probably want to comment out the _Add_DataNode() function as it will keep adding a Data node section to the end of your sheet, it was basically just an example.

Thank you for the example, Subz! I love the varied uses of the DOM object directly. Combined with examining how the XML UDF internally handles XML objects I think I have pretty much everything I need code-wise for dealing with the XML DOM. From what I can tell it's going to be best if I use For...In loops combined with With...EndWith statements (wherever it makes sense, of course) for readability and shorter coding. If that's incorrect, or you have additional thoughts, please feel free to let me know!

 

Earthshine - I'm not sure what you posted since I didn't get to read it before you edited but thank you for your reply, as well!

 

Thank you, all. I appreciate you taking time out of your day to provide me with assistance!

[Subz]

You may want to have a look at a post I created about a week ago, it may also help, I actually tried using XML udf but because my xml like yours didn't have any id attributes I ended up using DOM object directly and getting information directly from msdn and other sites on the code.  You'll notice I ended up using createNode method but actually prefer the createElement method I posted for you, so will probably change my code to reflect this.

 

[Jokerman]

Thanks for the link, Subz. I went over and read through it already. I like the straight XML DOM use. I ended up using the XML UDF almost exclusively for my solution (except for the $oNode_enum.Text on line 12 - it was just too easy lol):

 

expandcollapse popup

Local $aNodeNames = ["field1","field2","field3","field4","field5","field6","field7","field8","optionSetting"]
Local $aData = [["name12","email","choice1","choice2","choice3","choice4","choice5","choice7","true"], _
["name13","email","choice1","choice2","choice3","choice4","choice5","choice7","true"], _
["name14","email","choice1","choice2","choice3","choice4","choice5","choice7","true"]]

;~ BEGIN Preparation
Local $oNodes_coll = _XML_SelectNodes($oXMLDoc, "//Data[dataType='T1']/dataDetails/friendlyName")
Local $iNodeCount = @extended
Local $aDataTemp[$iNodeCount][UBound($aData, 2)]
Local $iNum = 0
For $oNode_enum In $oNodes_coll
  $aDataTemp[$iNum][0] = $oNode_enum.Text
  $iNum += 1
Next
_ArrayAdd($aDataTemp, $aData)
_ArrayDedupe($aDataTemp)
_ArrayDelete($aDataTemp, "0-" & $iNodeCount - 1)
$aData = $aDataTemp
$aDataTemp = 0
;~ END Preparation

;~ BEGIN Add data to XML
Local $aXSI[1][2] = [['xsi:type','Type1']]
For $iOuter = 0 To UBound($aData, 1) - 1
    _XML_CreateChildWAttr($oXMLDoc, '/ArrayOfData', 'Data')
    _XML_CreateChildWAttr($oXMLDoc, '//Data[last()]', 'Enabled', Default, 'true')
    _XML_CreateChildWAttr($oXMLDoc, '//Data[last()]', 'dataType', Default, 'T1')
    _XML_CreateChildWAttr($oXMLDoc, '//Data[last()]', 'dataDetails', $aXSI)
    For $iInner = 0 To UBound($aNodeNames) - 2
        _XML_CreateChildWAttr($oXMLDoc, '//Data[last()]/dataDetails', $aNodeNames[$iInner], Default, $aData[$iOuter][$iInner])
    Next
    _XML_CreateChildWAttr($oXMLDoc, '//Data[last()]', $aNodeNames[UBound($aNodeNames) - 1], Default, $aData[$iOuter][UBound($aNodeNames) - 1])
Next
;~ END Add data to XML

;~ BEGIN Cleanup
Local $sXmlAfterTidy = _XML_Tidy($oXMLDoc, "utf-8", False)
$sXMLAfterTidy = StringRegExpReplace($sXMLAfterTidy, "\t", "  ")
$sXmlAfterTidy = StringRegExpReplace($sXmlAfterTidy, "(\w+)(/>)", "\1 \2")
;~ END Cleanup


;~ My array dedupe func - _ArrayDedupe(ByRef $aArray[, $bReplace = False])
;~ Deduplicates an array based on the values in $aData[$x][0] keeping either first or last depending on $bReplace
Func _ArrayDedupe(ByRef $aArray, $bReplace = False)
    Local $i = 0
    Local $a1 = $aArray
    Do
        Local $a2 = _ArrayFindAll($a1, $a1[$i][0])
        If $bReplace Then
            $a2[0] = UBound($a2) - 1
        Else
            _ArrayPush($a2, "0", 1)
        EndIf
        _ArrayDelete($a1, $a2 )
        $i += 1
    Until $i >= UBound($a1)
    $aArray = $a1
EndFunc

I prefer to stay away from COM stuff unless I have to because I've run into code processors in the past (obfuscators and such) that have a hard time deciphering COM objects using ".Method" notation. While I'm past the point of using an obfuscator (for many reasons noted on this forum) I may be running other things on my code like codescanner or codecrypter, and I prefer to avoid as many issues as possible.

Feel free to comment on my code. I'm always looking for the best/most efficient way to do things so if anyone has constructive criticism please don't hesitate to let me know.

Edit - I almost forgot, don't try to use my _XML_Tidy as is - I extended that function in my XML.au3 to include an additional parameter for whether to keep the "standalone" declaration that's automatically added. The program I'm exporting to doesn't like that parameter. 

Edited April 11, 2018 by Jokerman
Additional information