data search in multi file

[trescon]

Good evening, I would have to ask you a little help.

So I have 4 files to manage, in the sense that these files are there 'a common field for all that allows me to identify them so that they can be merged into a single file.

The files consist of data in csv format to a file length of even 120,000 records.

I then I need suggestions to find the best way to make an effective and rapid search through all records of the four files in order to merge the data of the four files with the same unique code in a single file.

Does anyone have suggestions?

thanks

Alberto

Edited October 20, 2012 by trescon

[water]

I would suggest to import the CSV files into a database. They are made for searching and merging big amount of data.

[trescon]

Yes, of course, but I want to do it with autoit.

I want to do with autoit also because 'I must try some data on that large file, I do not always handle everything after the research that I make records to manage than this enso large files that are not more' than 400/500.

My problem is' to know what is a reasonable algorithm to search for data within a file loaded for better management in an array.

thanks

Alberto

[AZJIO]

[water]

Are the data files sorted on the common field?

[trescon]

I'm sorry, what do you mean by common ground?

[water]

You are talking about that "there is a common field for all that allows me to identify them". Are the data records sorted by this field?

[trescon]

You are talking about that "there is a common field for all that allows me to identify them". Are the data records sorted by this field?

not, this field is a "code" number with a tendency to grow with every new product added to the range (the range goes from code "0000000" to "110000").

this area code and 'the only field in common between 4 archives de vo manage, to manage in which I intend to do some research based on the "code".

I take from 4 files, descriptions, quantities, stocks and prices, each archive can 'contain one or more of these information.

Archives (4) do not all have the same length, both of records of different fields.

The largest archive and 'the one that contains the registry of the article, the other files instead may contain important dates and stocks that I need to discrimination.

Files can be some csv or excel format.

I hope I have cleared things up a bit.

thanks

Alberto

[water]

If the files you want to process can be in Excel format you could use

[trescon]

The files are in CSV format and not in excel.

[jchd]

You can import your 4 CSV files into 4 SQLite tables (an easy-to-use RDBMS engine directly manageable using AutoIt). Doing so is easy thanks to the command-line executable, SQLite3.exe.

Once all data is loaded in a database, you can create the final table consisting of all the columns you need and use SQL to merge data there according to your needs and rules. This (presumably a one-time task) will be easier if you use a good SQLite DB manager, like SQLite Expert Personal (freeware version).

When you have the final table(s) ready, you can use AutoIt to routinely manipulate/query/update your SQLite database.

[water]

In post #8 you statet: "Files can be some csv or excel format."

Anyway, as jchd and I suggested for "data analysis" use a SQL database.

[trescon]

You can import your 4 CSV files into 4 SQLite tables (an easy-to-use RDBMS engine directly manageable using AutoIt). Doing so is easy thanks to the command-line executable, SQLite3.exe.

Once all data is loaded in a database, you can create the final table consisting of all the columns you need and use SQL to merge data there according to your needs and rules. This (presumably a one-time task) will be easier if you use a good SQLite DB manager, like SQLite Expert Personal (freeware version).

When you have the final table(s) ready, you can use AutoIt to routinely manipulate/query/update your SQLite database.

In post #8 you statet: "Files can be some csv or excel format."

Anyway, as jchd and I suggested for "data analysis" use a SQL database.

Good evening, meanwhile, thanks for your suggestions, but they are too complex for my knowledge, and perhaps still are too "powerful" than it is to me.

I just have to find within all four files, the data corresponding to each code in my possession (present in one-fifth rows) and combine them all on one record, but in practice I do four surveys in four rows and when I find equality with my data i copy all the data that is connected to a common file.

I hope I was clear and I hope some suggestions on your part.

Both for the best technique to do the research is to create any INDEXES if necessary.

Best Regards

[water]

If a database is too complex then maybe the problem is too complex as well

To write good and manageable code you need an algorithm that is appropriate for your problem. To handle half a million records you need more then just a few loops reading through the files.

I would suggest to import the CSV files into a database and then do all the necessary processing.

Just my 0.02$ worth.

[jchd]

Just like Water I also stand by my answer. As you can see we're willing to help you learning how to perform elegantly and powerfully this kind of task. Not only you'll learn a lot but experience shows that over time, needs grow and get more complex. Hence putting this on solid feet from the ground up is looking forward in time.

But then, we can't (and won't) force your choice.