Jump to content

Stuck on concept need community help to start coding Server<>Client app


Recommended Posts

I created an app using autoit that would automate tasks in my windows environment. Mostly manipulating the file system etc, but most importantly I needed to capture information on each machine and store the found information on the server so that I can then manually analyze it. I got about 300 machines writing to multiple ini files and log files but along the way I learned a very good lesson. Because of how windows locks on to files I found that I'd often loose information and or would have inconsistent data. The more systems I'd have writing to the files the more I would see this. My application stays running on each client, on a sleep timer, I have a random sleep timer set yet I still run into some trouble with clients writing at the same time.

Since I learned this the hard way I wanted to take on a new challenge but I am stuck. The reason why I created .ini files was for me to be able to look at the data on my server from each client and perform different tasks based on the data I found stored on the server. So I must have that two way communication available at all times. I understand that using SQLLite for multiple writes is plagued by the same file locking issues as .ini files etc....

I am looking for suggestions. I am looking for something that doesn't require installation on the client. Sending flat files like .dll's or .exe files is fine as long as it doesn't require installation on the clients.

I started to look into FireBird but it seems like this requires installation on the client for it to work.

I started to look into SQLLite but I have a few concerns and really unsure on how to proceed. My questions are

1. I would want 1 SQLLite db file, how will I circumvent the file locking issues if I am continuously reading and writing to a single file?

Even if I figure out a way to do some sort of queuing through some sort of xmlhttp send to a php script that writes to a sqlLite file, I'll still be constantly reading it.

Thanks for the ideas and taking the time to read this.

Link to comment
Share on other sites

Not sure that this will help or not...since you are going to be reading the files manually, there is no way you are going to keep up with that amount of data. So, maybe you write the files, one for each machine, and close it each day, sending it to your main repository. Write another script to open each of the written files into whatever format you need to reach the conclusion you are looking for. The next day you will have a lot of data to parse and you may get what you need. You then add that to the main file/files, and in that way you are not working on live/opened/locked files?

All by me:

"Sometimes you have to go back to where you started, to get to where you want to go." 

"Everybody catches up with everyone, eventually" 

"As you teach others, you are really teaching yourself."

From my dad

"Do not worry about yesterday, as the only thing that you can control is tomorrow."

 

WindowsError.gif

WIKI | Tabs; | Arrays; | Strings | Wiki Arrays | How to ask a Question | Forum Search | FAQ | Tutorials | Original FAQ | ONLINE HELP | UDF's Wiki | AutoIt PDF

AutoIt Snippets | Multple Guis | Interrupting a running function | Another Send

StringRegExp | StringRegExp Help | RegEXTester | REG TUTOR | Reg TUTOT 2

AutoItSetOption | Macros | AutoIt Snippets | Wrapper | Autoit  Docs

SCITE | SciteJump | BB | MyTopics | Programming | UDFs | AutoIt 123 | UDFs Form | UDF

Learning to script | Tutorials | Documentation | IE.AU3 | Games? | FreeSoftware | Path_Online | Core Language

Programming Tips

Excel Changes

ControlHover.UDF

GDI_Plus

Draw_On_Screen

GDI Basics

GDI_More_Basics

GDI Rotate

GDI Graph

GDI  CheckExistingItems

GDI Trajectory

Replace $ghGDIPDll with $__g_hGDIPDll

DLL 101?

Array via Object

GDI Swimlane

GDI Plus French 101 Site

GDI Examples UEZ

GDI Basic Clock

GDI Detection

Ternary operator

Link to comment
Share on other sites

SQLite single database accessed by multiple machines: big no no, with one exception:

if you manage your database files, whatever the format you choose, you must manage the lock protection.

one way is working fine for me is (client algo):

1) hi, i'm the client. i want to write to a log file located on the server: serversharelog.txt

2) i'm checking if the file is in use by any of my client friends. is the lock file serversharelog.txt.lck exist?

3) wait until the lock file does not exist.

4) hey, the lock file is gone! i can write now! but wait, i must lock the file for myself first.

5) create the lock file serversharelog.txt.lck

6) write to the log file serversharelog.txt

7) delete the lock file serversharelog.txt.lck

8) bye!

now, if all your clients use same algo, you're good.

you can add to it: the lock file may contain the host name or process id of the host/process who locked it, so step 3 would be:

3) loop until lock file does not exist, or some timeout reached. if timeout, then query the lock file to see if the host/process are still alive; if not, delete the lock.

PS you do NOT need 2-way communication. the server needs no code running.

or am i misunderstanding your intention?

Signature - my forum contributions:

Spoiler

UDF:

LFN - support for long file names (over 260 characters)

InputImpose - impose valid characters in an input control

TimeConvert - convert UTC to/from local time and/or reformat the string representation

AMF - accept multiple files from Windows Explorer context menu

DateDuration -  literal description of the difference between given dates

Apps:

Touch - set the "modified" timestamp of a file to current time

Show For Files - tray menu to show/hide files extensions, hidden & system files, and selection checkboxes

SPDiff - Single-Pane Text Diff

 

Link to comment
Share on other sites

I would split your projects into two parts: data collection and data processing.

I assume you are in a Windows network environment and all computers write their data to the same share. Create a file for each computer with a unique name (@ComputerName) on this share and let the client store its data there.

Later let the processing script loop through all the data files and do whatever is needed.

My UDFs and Tutorials:

Spoiler

UDFs:
Active Directory (NEW 2022-02-19 - Version 1.6.1.0) - Download - General Help & Support - Example Scripts - Wiki
ExcelChart (2017-07-21 - Version 0.4.0.1) - Download - General Help & Support - Example Scripts
OutlookEX (2021-11-16 - Version 1.7.0.0) - Download - General Help & Support - Example Scripts - Wiki
OutlookEX_GUI (2021-04-13 - Version 1.4.0.0) - Download
Outlook Tools (2019-07-22 - Version 0.6.0.0) - Download - General Help & Support - Wiki
PowerPoint (2021-08-31 - Version 1.5.0.0) - Download - General Help & Support - Example Scripts - Wiki
Task Scheduler (NEW 2022-07-28 - Version 1.6.0.1) - Download - General Help & Support - Wiki

Standard UDFs:
Excel - Example Scripts - Wiki
Word - Wiki

Tutorials:
ADO - Wiki
WebDriver - Wiki

 

Link to comment
Share on other sites

I would split your projects into two parts: data collection and data processing.

I assume you are in a Windows network environment and all computers write their data to the same share. Create a file for each computer with a unique name (@ComputerName) on this share and let the client store its data there.

Later let the processing script loop through all the data files and do whatever is needed.

Yea, what @water said...

All by me:

"Sometimes you have to go back to where you started, to get to where you want to go." 

"Everybody catches up with everyone, eventually" 

"As you teach others, you are really teaching yourself."

From my dad

"Do not worry about yesterday, as the only thing that you can control is tomorrow."

 

WindowsError.gif

WIKI | Tabs; | Arrays; | Strings | Wiki Arrays | How to ask a Question | Forum Search | FAQ | Tutorials | Original FAQ | ONLINE HELP | UDF's Wiki | AutoIt PDF

AutoIt Snippets | Multple Guis | Interrupting a running function | Another Send

StringRegExp | StringRegExp Help | RegEXTester | REG TUTOR | Reg TUTOT 2

AutoItSetOption | Macros | AutoIt Snippets | Wrapper | Autoit  Docs

SCITE | SciteJump | BB | MyTopics | Programming | UDFs | AutoIt 123 | UDFs Form | UDF

Learning to script | Tutorials | Documentation | IE.AU3 | Games? | FreeSoftware | Path_Online | Core Language

Programming Tips

Excel Changes

ControlHover.UDF

GDI_Plus

Draw_On_Screen

GDI Basics

GDI_More_Basics

GDI Rotate

GDI Graph

GDI  CheckExistingItems

GDI Trajectory

Replace $ghGDIPDll with $__g_hGDIPDll

DLL 101?

Array via Object

GDI Swimlane

GDI Plus French 101 Site

GDI Examples UEZ

GDI Basic Clock

GDI Detection

Ternary operator

Link to comment
Share on other sites

water's advice is good, provided there is a long enough period of time when only the "collector" is accessing the files, because if the "collector" tries that while the "processor" is working, you come to the same problem.

Signature - my forum contributions:

Spoiler

UDF:

LFN - support for long file names (over 260 characters)

InputImpose - impose valid characters in an input control

TimeConvert - convert UTC to/from local time and/or reformat the string representation

AMF - accept multiple files from Windows Explorer context menu

DateDuration -  literal description of the difference between given dates

Apps:

Touch - set the "modified" timestamp of a file to current time

Show For Files - tray menu to show/hide files extensions, hidden & system files, and selection checkboxes

SPDiff - Single-Pane Text Diff

 

Link to comment
Share on other sites

The idea would be to write the file with computer name and date stamp. Never writing to the same file. Once the file has been parsed, then it can be deleted. Your only going to be keep about 3 files per computer and your going to retrieve the data from a closed file.

All by me:

"Sometimes you have to go back to where you started, to get to where you want to go." 

"Everybody catches up with everyone, eventually" 

"As you teach others, you are really teaching yourself."

From my dad

"Do not worry about yesterday, as the only thing that you can control is tomorrow."

 

WindowsError.gif

WIKI | Tabs; | Arrays; | Strings | Wiki Arrays | How to ask a Question | Forum Search | FAQ | Tutorials | Original FAQ | ONLINE HELP | UDF's Wiki | AutoIt PDF

AutoIt Snippets | Multple Guis | Interrupting a running function | Another Send

StringRegExp | StringRegExp Help | RegEXTester | REG TUTOR | Reg TUTOT 2

AutoItSetOption | Macros | AutoIt Snippets | Wrapper | Autoit  Docs

SCITE | SciteJump | BB | MyTopics | Programming | UDFs | AutoIt 123 | UDFs Form | UDF

Learning to script | Tutorials | Documentation | IE.AU3 | Games? | FreeSoftware | Path_Online | Core Language

Programming Tips

Excel Changes

ControlHover.UDF

GDI_Plus

Draw_On_Screen

GDI Basics

GDI_More_Basics

GDI Rotate

GDI Graph

GDI  CheckExistingItems

GDI Trajectory

Replace $ghGDIPDll with $__g_hGDIPDll

DLL 101?

Array via Object

GDI Swimlane

GDI Plus French 101 Site

GDI Examples UEZ

GDI Basic Clock

GDI Detection

Ternary operator

Link to comment
Share on other sites

If you want to make sure that the collector has finished writing to the data file before the processor can start I suggest to use a lock file.

The collector creates a file named @ComputerName & ".lock" before writing to @ComputerName & ".data". When finished the lock file is deleted by the collector. So if the processor finds a lock file for a computer he has to wait or process another computer until the lock file is deleted.

My UDFs and Tutorials:

Spoiler

UDFs:
Active Directory (NEW 2022-02-19 - Version 1.6.1.0) - Download - General Help & Support - Example Scripts - Wiki
ExcelChart (2017-07-21 - Version 0.4.0.1) - Download - General Help & Support - Example Scripts
OutlookEX (2021-11-16 - Version 1.7.0.0) - Download - General Help & Support - Example Scripts - Wiki
OutlookEX_GUI (2021-04-13 - Version 1.4.0.0) - Download
Outlook Tools (2019-07-22 - Version 0.6.0.0) - Download - General Help & Support - Wiki
PowerPoint (2021-08-31 - Version 1.5.0.0) - Download - General Help & Support - Example Scripts - Wiki
Task Scheduler (NEW 2022-07-28 - Version 1.6.0.1) - Download - General Help & Support - Wiki

Standard UDFs:
Excel - Example Scripts - Wiki
Word - Wiki

Tutorials:
ADO - Wiki
WebDriver - Wiki

 

Link to comment
Share on other sites

using "collector-processor" method has one disadvantage - data is available for analysis after the collector is done. using real-time single-file lock/unlock method gives you real-time data to analyse, you don't have to wait for the next day.

Signature - my forum contributions:

Spoiler

UDF:

LFN - support for long file names (over 260 characters)

InputImpose - impose valid characters in an input control

TimeConvert - convert UTC to/from local time and/or reformat the string representation

AMF - accept multiple files from Windows Explorer context menu

DateDuration -  literal description of the difference between given dates

Apps:

Touch - set the "modified" timestamp of a file to current time

Show For Files - tray menu to show/hide files extensions, hidden & system files, and selection checkboxes

SPDiff - Single-Pane Text Diff

 

Link to comment
Share on other sites

If you need real-time data then I suggest using a database. Add a time-stamp to every record so you know how current and complete the records of a single PC are. If the collector is updating the data once a day you might process data of today and yesterday for the same computer. Or the collector puts everything into a single transaction and does a commit when all data has been collected.

It all depends on your requirements.

My UDFs and Tutorials:

Spoiler

UDFs:
Active Directory (NEW 2022-02-19 - Version 1.6.1.0) - Download - General Help & Support - Example Scripts - Wiki
ExcelChart (2017-07-21 - Version 0.4.0.1) - Download - General Help & Support - Example Scripts
OutlookEX (2021-11-16 - Version 1.7.0.0) - Download - General Help & Support - Example Scripts - Wiki
OutlookEX_GUI (2021-04-13 - Version 1.4.0.0) - Download
Outlook Tools (2019-07-22 - Version 0.6.0.0) - Download - General Help & Support - Wiki
PowerPoint (2021-08-31 - Version 1.5.0.0) - Download - General Help & Support - Example Scripts - Wiki
Task Scheduler (NEW 2022-07-28 - Version 1.6.0.1) - Download - General Help & Support - Wiki

Standard UDFs:
Excel - Example Scripts - Wiki
Word - Wiki

Tutorials:
ADO - Wiki
WebDriver - Wiki

 

Link to comment
Share on other sites

I was thinking about a solution along the lines of Water's.

Each client would keep a timestamped log and send it to the server in an hourly fashion (configurable) with additional data so as to identify which client sent any given log file.

You can then have two approaches, either create a master log file by using the received timestamps plus client data in one big file (performance issues) or code a GUI that would list those log files per client and per timestamp with some neat filters for dates.

I think this eliminates the need of checking file locks, as each client would post the log for you on your server and you would simply read them.

Using a database with multi-user support would be my next approach.

Link to comment
Share on other sites

  • 4 weeks later...

Is it not best to set up a message queue, so that all the machines post to that queue, and let the queue handler be solely responsible for writing to the file? Alternatively as suggested above, and simpler if you aren't a programmer, just use a database.

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

  • Recently Browsing   0 members

    • No registered users viewing this page.
×
×
  • Create New...