Any extra Ideas

satcom-neil · Post by **satcom-neil** » Mon Oct 15, 2007 11:10 am

Hey Everyone been working on a python program that reads though some pretty big data files 100MB is the smallest. All are tab delineated so it's quite a bit of data to parse though. I've already setup filters to narrow the about down which speeds it up. Just wanting some others ideas that might help speed this up in case the user want to read the whole file. Right now it takes about 30 mins for just a 138MB file and I know there are 500MB files to read on a bad day. I was hope maybe you guys could come up with any idea to speed up reading?

Neil

riddlebox · Post by **riddlebox** » Wed Nov 07, 2007 6:29 pm

I am working on an app that I need to read the number of lines in a txt file and and then take that number and replace a certain number in another text file? Any ideas on how I would do that?

satcom-neil · Post by **satcom-neil** » Fri Nov 09, 2007 9:09 am

riddle

I've gotten it to run the 138MB file in about 10-14 mins just by reading the whole file into memory. I've been thinking about putting python to work by using "For line in file" Or using sql to speed it up more since a DB would be optimum.

If you come up with better let me know

My big problem has been displaying the data in a useful manner.

Neil

riddlebox · Post by **riddlebox** » Thu Nov 15, 2007 8:04 pm

I think I have found out how to work with my string the way I need to, with a file as big as what you are dealing with, would reading the file line by line even work for you?

satcom-neil · Post by **satcom-neil** » Fri Nov 16, 2007 7:49 am

That did work it was about the same speed it's just a ton of data I'm reading 30MB file about a min and a half. Truly I need to get a hold of the guys that did the first program he used MS SQL to get around reading the files just it's not easy to get in touch with him. I've got another project that I'm going to really start on shortly for the N770 - the New one when they get it released.

satcom-neil · Post by **satcom-neil** » Fri Nov 16, 2007 8:14 am

That did work it was about the same speed it's just a ton of data I'm reading 30MB file about a min and a half. Truly I need to get a hold of the guys that did the first program he used MS SQL to get around reading the files just it's not easy to get in touch with him. I've got another project that I'm going to really start on shortly for the N770 - the New one when they get it released.