The number 4096 can burn in hell.

[jchd]

I can think of ways to remove that limitation but they aren't really trivial to implement. The trivial ways to implement it all impact performance. The reason a static line size is used is so that constant allocations and deallocations don't occur.

Got that, thank you. Would a command-line switch or directive fit the bill so that the 4000 default static buffer fit most uses and uncommon scripts can be processed as well?

[Jon]

Actually, the line size is only used during the phase where text is turned into tokens. There's no limit on the token vectors so it might be worth checking out if the line size limitation is mostly laziness and see if we can swap it out for something. I need to have a quick look at the source to be sure.

[jchd]

Actually, the line size is only used during the phase where text is turned into tokens. There's no limit on the token vectors so it might be worth checking out if the line size limitation is mostly laziness and see if we can swap it out for an AString type or something.

Thanks for looking. But I wouldn't want that a relatively uncommon usage would need too much deep change. If it can be easily switched to dynamic alloc on special demand and keep on using the static area for regular use, then that's fine. If not, then there are always an alternative available by loading the data from a file, even if it's a little less convenient in some situations.

[Valik]

Actually, the line size is only used during the phase where text is turned into tokens. There's no limit on the token vectors so it might be worth checking out if the line size limitation is mostly laziness and see if we can swap it out for something. I need to have a quick look at the source to be sure.

I think it probably started out as laziness (you were a lazy...er bastard back then) but it also has it's performance advantages as well. There are two main options that I can see.

• Pre-scan the file looking for the longest line and then allocate a buffer of that size which is then used on a second pass through the file to actually read and parse the lines.
• Allocate a reasonably sized buffer to begin with and re-allocate a new buffer every time the line length exceeds the current buffer. This could potentially involve dozens of deallocations/allocations depending on the way the file is made up. Front-loading the longest line near the beginning would make the allocations a non-issue since it would happen twice.

I would probably opt for the second approach. Also, in case its not obvious the buffer would be static although that would then need a clever solution to free the buffer once it's well and truly no longer needed.

[Jon]

Had a look. Could easily change the line size to something more reasonable like 64k without impacting perf (as far as I could see), but ideally it needs all the strings converting to our internal string class which resizes itself quite happily. But unfortunately that wouldn't help at the moment as the ulitmate problem is our file/line reading code that requires a preallocated buffer size to function. This is also the reason that this ticket hasn't been closed #682. I recently did a lot of work in preperation for getting that ticket closed but its probably best we do that first before messing around with other code.

[jchd]

Had a look. Could easily change the line size to something more reasonable like 64k without impacting perf (as far as I could see), but ideally it needs all the strings converting to our internal string class which resizes itself quite happily. But unfortunately that wouldn't help at the moment as the ulitmate problem is our file/line reading code that requires a preallocated buffer size to function. This is also the reason that this ticket hasn't been closed #682. I recently did a lot of work in preperation for getting that ticket closed but its probably best we do that first before messing around with other code.

Jon, Jason and all the team,

Warm thanks again for your care. I sincerely didn't want to make a big fuss of this: the 4K subject simply remind me of that point. This isn't a high priority thing. Only you --having a clear view of the development directions and priorities-- can decide what needs [r]evolution.

[Jon]

Warm thanks again for your care. I sincerely didn't want to make a big fuss of this: the 4K subject simply remind me of that point. This isn't a high priority thing. Only you --having a clear view of the development directions and priorities-- can decide what needs [r]evolution.

Current beta allows the _ continuation character work around this issue now. Eventually I'll make it so that a single line is also unrestricted but this should do nicely for now.

[jchd]

Thanks Jon, that's making AutoIt a big-caliber tool! I bet the continuation already fits any reasonable need.