vortexed Posted June 3, 2013 Share Posted June 3, 2013 I created an app using autoit that would automate tasks in my windows environment. Mostly manipulating the file system etc, but most importantly I needed to capture information on each machine and store the found information on the server so that I can then manually analyze it. I got about 300 machines writing to multiple ini files and log files but along the way I learned a very good lesson. Because of how windows locks on to files I found that I'd often loose information and or would have inconsistent data. The more systems I'd have writing to the files the more I would see this. My application stays running on each client, on a sleep timer, I have a random sleep timer set yet I still run into some trouble with clients writing at the same time. Since I learned this the hard way I wanted to take on a new challenge but I am stuck. The reason why I created .ini files was for me to be able to look at the data on my server from each client and perform different tasks based on the data I found stored on the server. So I must have that two way communication available at all times. I understand that using SQLLite for multiple writes is plagued by the same file locking issues as .ini files etc.... I am looking for suggestions. I am looking for something that doesn't require installation on the client. Sending flat files like .dll's or .exe files is fine as long as it doesn't require installation on the clients. I started to look into FireBird but it seems like this requires installation on the client for it to work. I started to look into SQLLite but I have a few concerns and really unsure on how to proceed. My questions are 1. I would want 1 SQLLite db file, how will I circumvent the file locking issues if I am continuously reading and writing to a single file? Even if I figure out a way to do some sort of queuing through some sort of xmlhttp send to a php script that writes to a sqlLite file, I'll still be constantly reading it. Thanks for the ideas and taking the time to read this. Link to comment Share on other sites More sharing options...
nitekram Posted June 3, 2013 Share Posted June 3, 2013 Not sure that this will help or not...since you are going to be reading the files manually, there is no way you are going to keep up with that amount of data. So, maybe you write the files, one for each machine, and close it each day, sending it to your main repository. Write another script to open each of the written files into whatever format you need to reach the conclusion you are looking for. The next day you will have a lot of data to parse and you may get what you need. You then add that to the main file/files, and in that way you are not working on live/opened/locked files? 2¢ All by me:"Sometimes you have to go back to where you started, to get to where you want to go." "Everybody catches up with everyone, eventually" "As you teach others, you are really teaching yourself." From my dad "Do not worry about yesterday, as the only thing that you can control is tomorrow." WIKI | Tabs; | Arrays; | Strings | Wiki Arrays | How to ask a Question | Forum Search | FAQ | Tutorials | Original FAQ | ONLINE HELP | UDF's Wiki | AutoIt PDF AutoIt Snippets | Multple Guis | Interrupting a running function | Another Send StringRegExp | StringRegExp Help | RegEXTester | REG TUTOR | Reg TUTOT 2 AutoItSetOption | Macros | AutoIt Snippets | Wrapper | Autoit Docs SCITE | SciteJump | BB | MyTopics | Programming | UDFs | AutoIt 123 | UDFs Form | UDF Learning to script | Tutorials | Documentation | IE.AU3 | Games? | FreeSoftware | Path_Online | Core Language Programming Tips Excel Changes ControlHover.UDF GDI_Plus Draw_On_Screen GDI Basics GDI_More_Basics GDI Rotate GDI Graph GDI CheckExistingItems GDI Trajectory Replace $ghGDIPDll with $__g_hGDIPDll DLL 101? Array via Object GDI Swimlane GDI Plus French 101 Site GDI Examples UEZ GDI Basic Clock GDI Detection Ternary operator Link to comment Share on other sites More sharing options...
orbs Posted June 3, 2013 Share Posted June 3, 2013 SQLite single database accessed by multiple machines: big no no, with one exception: if you manage your database files, whatever the format you choose, you must manage the lock protection. one way is working fine for me is (client algo): 1) hi, i'm the client. i want to write to a log file located on the server: serversharelog.txt 2) i'm checking if the file is in use by any of my client friends. is the lock file serversharelog.txt.lck exist? 3) wait until the lock file does not exist. 4) hey, the lock file is gone! i can write now! but wait, i must lock the file for myself first. 5) create the lock file serversharelog.txt.lck 6) write to the log file serversharelog.txt 7) delete the lock file serversharelog.txt.lck 8) bye! now, if all your clients use same algo, you're good. you can add to it: the lock file may contain the host name or process id of the host/process who locked it, so step 3 would be: 3) loop until lock file does not exist, or some timeout reached. if timeout, then query the lock file to see if the host/process are still alive; if not, delete the lock. PS you do NOT need 2-way communication. the server needs no code running. or am i misunderstanding your intention? Signature - my forum contributions: Spoiler UDF: LFN - support for long file names (over 260 characters) InputImpose - impose valid characters in an input control TimeConvert - convert UTC to/from local time and/or reformat the string representation AMF - accept multiple files from Windows Explorer context menu DateDuration - literal description of the difference between given dates Apps: Touch - set the "modified" timestamp of a file to current time Show For Files - tray menu to show/hide files extensions, hidden & system files, and selection checkboxes SPDiff - Single-Pane Text Diff Link to comment Share on other sites More sharing options...
water Posted June 3, 2013 Share Posted June 3, 2013 I would split your projects into two parts: data collection and data processing. I assume you are in a Windows network environment and all computers write their data to the same share. Create a file for each computer with a unique name (@ComputerName) on this share and let the client store its data there. Later let the processing script loop through all the data files and do whatever is needed. My UDFs and Tutorials: Spoiler UDFs:Active Directory (NEW 2022-02-19 - Version 1.6.1.0) - Download - General Help & Support - Example Scripts - WikiExcelChart (2017-07-21 - Version 0.4.0.1) - Download - General Help & Support - Example ScriptsOutlookEX (2021-11-16 - Version 1.7.0.0) - Download - General Help & Support - Example Scripts - WikiOutlookEX_GUI (2021-04-13 - Version 1.4.0.0) - DownloadOutlook Tools (2019-07-22 - Version 0.6.0.0) - Download - General Help & Support - WikiPowerPoint (2021-08-31 - Version 1.5.0.0) - Download - General Help & Support - Example Scripts - WikiTask Scheduler (NEW 2022-07-28 - Version 1.6.0.1) - Download - General Help & Support - Wiki Standard UDFs:Excel - Example Scripts - WikiWord - Wiki Tutorials:ADO - WikiWebDriver - Wiki Link to comment Share on other sites More sharing options...
nitekram Posted June 3, 2013 Share Posted June 3, 2013 I would split your projects into two parts: data collection and data processing.I assume you are in a Windows network environment and all computers write their data to the same share. Create a file for each computer with a unique name (@ComputerName) on this share and let the client store its data there.Later let the processing script loop through all the data files and do whatever is needed.Yea, what @water said... 2¢ All by me:"Sometimes you have to go back to where you started, to get to where you want to go." "Everybody catches up with everyone, eventually" "As you teach others, you are really teaching yourself." From my dad "Do not worry about yesterday, as the only thing that you can control is tomorrow." WIKI | Tabs; | Arrays; | Strings | Wiki Arrays | How to ask a Question | Forum Search | FAQ | Tutorials | Original FAQ | ONLINE HELP | UDF's Wiki | AutoIt PDF AutoIt Snippets | Multple Guis | Interrupting a running function | Another Send StringRegExp | StringRegExp Help | RegEXTester | REG TUTOR | Reg TUTOT 2 AutoItSetOption | Macros | AutoIt Snippets | Wrapper | Autoit Docs SCITE | SciteJump | BB | MyTopics | Programming | UDFs | AutoIt 123 | UDFs Form | UDF Learning to script | Tutorials | Documentation | IE.AU3 | Games? | FreeSoftware | Path_Online | Core Language Programming Tips Excel Changes ControlHover.UDF GDI_Plus Draw_On_Screen GDI Basics GDI_More_Basics GDI Rotate GDI Graph GDI CheckExistingItems GDI Trajectory Replace $ghGDIPDll with $__g_hGDIPDll DLL 101? Array via Object GDI Swimlane GDI Plus French 101 Site GDI Examples UEZ GDI Basic Clock GDI Detection Ternary operator Link to comment Share on other sites More sharing options...
orbs Posted June 3, 2013 Share Posted June 3, 2013 water's advice is good, provided there is a long enough period of time when only the "collector" is accessing the files, because if the "collector" tries that while the "processor" is working, you come to the same problem. Signature - my forum contributions: Spoiler UDF: LFN - support for long file names (over 260 characters) InputImpose - impose valid characters in an input control TimeConvert - convert UTC to/from local time and/or reformat the string representation AMF - accept multiple files from Windows Explorer context menu DateDuration - literal description of the difference between given dates Apps: Touch - set the "modified" timestamp of a file to current time Show For Files - tray menu to show/hide files extensions, hidden & system files, and selection checkboxes SPDiff - Single-Pane Text Diff Link to comment Share on other sites More sharing options...
nitekram Posted June 3, 2013 Share Posted June 3, 2013 The idea would be to write the file with computer name and date stamp. Never writing to the same file. Once the file has been parsed, then it can be deleted. Your only going to be keep about 3 files per computer and your going to retrieve the data from a closed file. 2¢ All by me:"Sometimes you have to go back to where you started, to get to where you want to go." "Everybody catches up with everyone, eventually" "As you teach others, you are really teaching yourself." From my dad "Do not worry about yesterday, as the only thing that you can control is tomorrow." WIKI | Tabs; | Arrays; | Strings | Wiki Arrays | How to ask a Question | Forum Search | FAQ | Tutorials | Original FAQ | ONLINE HELP | UDF's Wiki | AutoIt PDF AutoIt Snippets | Multple Guis | Interrupting a running function | Another Send StringRegExp | StringRegExp Help | RegEXTester | REG TUTOR | Reg TUTOT 2 AutoItSetOption | Macros | AutoIt Snippets | Wrapper | Autoit Docs SCITE | SciteJump | BB | MyTopics | Programming | UDFs | AutoIt 123 | UDFs Form | UDF Learning to script | Tutorials | Documentation | IE.AU3 | Games? | FreeSoftware | Path_Online | Core Language Programming Tips Excel Changes ControlHover.UDF GDI_Plus Draw_On_Screen GDI Basics GDI_More_Basics GDI Rotate GDI Graph GDI CheckExistingItems GDI Trajectory Replace $ghGDIPDll with $__g_hGDIPDll DLL 101? Array via Object GDI Swimlane GDI Plus French 101 Site GDI Examples UEZ GDI Basic Clock GDI Detection Ternary operator Link to comment Share on other sites More sharing options...
water Posted June 3, 2013 Share Posted June 3, 2013 If you want to make sure that the collector has finished writing to the data file before the processor can start I suggest to use a lock file. The collector creates a file named @ComputerName & ".lock" before writing to @ComputerName & ".data". When finished the lock file is deleted by the collector. So if the processor finds a lock file for a computer he has to wait or process another computer until the lock file is deleted. My UDFs and Tutorials: Spoiler UDFs:Active Directory (NEW 2022-02-19 - Version 1.6.1.0) - Download - General Help & Support - Example Scripts - WikiExcelChart (2017-07-21 - Version 0.4.0.1) - Download - General Help & Support - Example ScriptsOutlookEX (2021-11-16 - Version 1.7.0.0) - Download - General Help & Support - Example Scripts - WikiOutlookEX_GUI (2021-04-13 - Version 1.4.0.0) - DownloadOutlook Tools (2019-07-22 - Version 0.6.0.0) - Download - General Help & Support - WikiPowerPoint (2021-08-31 - Version 1.5.0.0) - Download - General Help & Support - Example Scripts - WikiTask Scheduler (NEW 2022-07-28 - Version 1.6.0.1) - Download - General Help & Support - Wiki Standard UDFs:Excel - Example Scripts - WikiWord - Wiki Tutorials:ADO - WikiWebDriver - Wiki Link to comment Share on other sites More sharing options...
orbs Posted June 3, 2013 Share Posted June 3, 2013 using "collector-processor" method has one disadvantage - data is available for analysis after the collector is done. using real-time single-file lock/unlock method gives you real-time data to analyse, you don't have to wait for the next day. Signature - my forum contributions: Spoiler UDF: LFN - support for long file names (over 260 characters) InputImpose - impose valid characters in an input control TimeConvert - convert UTC to/from local time and/or reformat the string representation AMF - accept multiple files from Windows Explorer context menu DateDuration - literal description of the difference between given dates Apps: Touch - set the "modified" timestamp of a file to current time Show For Files - tray menu to show/hide files extensions, hidden & system files, and selection checkboxes SPDiff - Single-Pane Text Diff Link to comment Share on other sites More sharing options...
water Posted June 3, 2013 Share Posted June 3, 2013 If you need real-time data then I suggest using a database. Add a time-stamp to every record so you know how current and complete the records of a single PC are. If the collector is updating the data once a day you might process data of today and yesterday for the same computer. Or the collector puts everything into a single transaction and does a commit when all data has been collected. It all depends on your requirements. My UDFs and Tutorials: Spoiler UDFs:Active Directory (NEW 2022-02-19 - Version 1.6.1.0) - Download - General Help & Support - Example Scripts - WikiExcelChart (2017-07-21 - Version 0.4.0.1) - Download - General Help & Support - Example ScriptsOutlookEX (2021-11-16 - Version 1.7.0.0) - Download - General Help & Support - Example Scripts - WikiOutlookEX_GUI (2021-04-13 - Version 1.4.0.0) - DownloadOutlook Tools (2019-07-22 - Version 0.6.0.0) - Download - General Help & Support - WikiPowerPoint (2021-08-31 - Version 1.5.0.0) - Download - General Help & Support - Example Scripts - WikiTask Scheduler (NEW 2022-07-28 - Version 1.6.0.1) - Download - General Help & Support - Wiki Standard UDFs:Excel - Example Scripts - WikiWord - Wiki Tutorials:ADO - WikiWebDriver - Wiki Link to comment Share on other sites More sharing options...
leansoli Posted June 3, 2013 Share Posted June 3, 2013 I was thinking about a solution along the lines of Water's. Each client would keep a timestamped log and send it to the server in an hourly fashion (configurable) with additional data so as to identify which client sent any given log file. You can then have two approaches, either create a master log file by using the received timestamps plus client data in one big file (performance issues) or code a GUI that would list those log files per client and per timestamp with some neat filters for dates. I think this eliminates the need of checking file locks, as each client would post the log for you on your server and you would simply read them. Using a database with multi-user support would be my next approach. Link to comment Share on other sites More sharing options...
JeremyBOB Posted July 2, 2013 Share Posted July 2, 2013 Is it not best to set up a message queue, so that all the machines post to that queue, and let the queue handler be solely responsible for writing to the file? Alternatively as suggested above, and simpler if you aren't a programmer, just use a database. Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now