Script abending on FTP

[VeeDub]

Hi,

After some advice on how to troubleshoot an issue I am having with a script.

I have a script that runs some external commands, then zips the resulting data and then copies the resulting archives via FTP to another location. The script runs nightly as a Scheduled Task on a Windows Server 2003 box.

Once a week the amount of data being copied is significantly greater than the other days, and this results in the FTP process taking more than 24 hours to complete, which means that the Scheduled Task tries to start another instance of the script before the current instance has completed.

The script uses singleton to ensure that extra instances of the script are not started when there is an instance already running.

The problem that I am observing is that once the script enters the FTP phase of processing, after the Task Scheduler attempts to start another instance of the script, the script will complete the FTP transfer of the file that it is currently processing - but then the script terminates without FTP'ing the remaining files.

This behaviour only occurs once the script reaches the FTP stage. The total processing time for the "weekly" job is 3 - 4 days, so the Task Scheduler attempts to start a new job a few times during the scripts processing - but it is not until the job is into the FTP step that it falls over. I don't see any error either, the task just ends.

Suggestions on how to troubleshoot would be appreciated.

thanks

VW

[PaulIA]

Suggestions on how to troubleshoot would be appreciated

Probably not what you're going to want to hear, but you'll probably have to do this the hard way. Put some lines of code in your script that output an "I am here" type of statements to a log so that you can see where the program is at when it's running. When it crashes, check the log file for the last successful write and start debugging from there.

If you're code isn't too long, you might want to post it here for us to take a look at.

[VeeDub]

Probably not what you're going to want to hear, but you'll probably have to do this the hard way. Put some lines of code in your script that output an "I am here" type of statements to a log so that you can see where the program is at when it's running. When it crashes, check the log file for the last successful write and start debugging from there.

If you're code isn't too long, you might want to post it here for us to take a look at.

I think if that's the approach I need to take, I'll try and reproduce the error in an FTP-only script and then post that. The existing script has a fair bit of code plus the external commands are site specific which means you can't just run the script as-is to reproduce the behaviour I am observing.

Thanks

VW

[Confuzzled]

Stepping back and looking at the larger picture:

I'd seriously be questioning your procedures and reasons for moving such large volumes of data across a FTP link if it takes over 24 hours.

Maybe you should be looking at local backup procedures or faster links, or processing your data before transmission. Are you zipping up the data before transmission?

Some code would be useful to assist with troubleshooting...

[VeeDub]

Stepping back and looking at the larger picture:

I'd seriously be questioning your procedures and reasons for moving such large volumes of data across a FTP link if it takes over 24 hours.

Maybe you should be looking at local backup procedures or faster links, or processing your data before transmission. Are you zipping up the data before transmission?

Some code would be useful to assist with troubleshooting...

@Confuzzled

It's a backup and there is a lot of data (there was approx 17GB compressed this week)

I do zip the data before transmission and then I copy it across a WAN link where I get 100K per sec. The time for the FTP doesn't really bother me, what does bother me is that the weekly script falls over in around the same place every week, there is a pattern so there is a reason.

I've extracted the FTP procedures from the full script and I've done a test on my computer and the FTP stand-alone doesn't fail, so it looks like the problem is either with the whole script or it has something to do with the other box that it is running it on.

I don't think it's reasonable for me to post the whole script, it's about 1,500 lines of code - and ask people to fix it. I think I should at least isolate the problem first, and then post the section that is causing the problem.

Thanks for your advice

[Confuzzled]

17Gb over a 100K link is going to take over 24 hours, regardless of what code you use to drive it.

Like I said, re-evaluate what you are doing.