Working with CSV files

I have 2 contact information CSV files that I need to merge into 1 file. One of the files is a 5000 record master and the other is a file generated from numerous keypunch personel with approximately 2000 records.

The basic processing model is as follows:

- load the master csv file
- load the manual entry contact file
- iterate through the manual file and search the master based in a key
- if not found, insert
- if found, perform an update
- generate a new output file in csv format

I could do this in Delphi in about 2 hours, but since I am trying to learn .NET and ASP.NET, I would like to attempt this small app. It will also give me the chance to work with unit testing using Nunit.NET.

Could someone give me some pointers for this app? Specifically, I am interested in the following information:

- What kind of data structure(s) should I load the csv files into?
- Performance isn't a big concern, but I would like to optimize it anyway.
- The code will be performing approx. 2000 searches against a 5000 record master
- Searches will be performed on more than 1 field at a time

- Can I do this app using .NET of will there be issues transferring the CSV files to the web server for processing?

- The application must browse the users computer for the two input files and save the output file back to the users computer. Since this application will only be run internally within the company, security is not an issue, but I'm not sure if there are other concerns I should have.

- Will the performance of a web application in this case be a big enough problem that I should just build a Win32 application. I would rather build a web app because I am hoping to build an entire web framework to handle security, logging, message, etc for all future application in my company.

Thanks,
Kevin MeierComing from a security standpoint, this paragraph you wrote worries me...

"- The application must browse the users computer for the two input files and save the output file back to the users computer. Since this application will only be run internally within the company, security is not an issue, but I'm not sure if there are other concerns I should have."

What do you mean by that?, the app would find these files on their hard-drive, or the client would upload/specify where to find them. I'd be kinda worried from a security standpoint if your application is searching around their hard-drive.

Do you have sql server? I'd probalby be loading these csv files into temporary tables and doing the inserts/updates there, then spit it back out to csv. That way you can just use sql to manage the operations.

Honestly though, this really has "Windows Application" written all over it.
Michael,

What I mean't by security not being a concern is that the user running the app will be company users to they need not be concerned with allowing access to their local drive from a corporate app.

You are correct in assuming that the client will be responsible for specifying the locations of the working csv files on their local disk and specifying the location to save the new file to.

As far as using a database... We are an Oracle shop so that is definately a consideration, however, I really want to use this as a opportunity to dive deeper into .NET and get more involved with unit testing (using .NET). So I guess you could say that I don't want to use a database for purely selfish reasons.

I agree with you that this looks and smells like a Win32 app, but once again, for selfish reasons, I want to experiement with web development because that is where most application in my company will be going anyway. This just gives me a chance to start realy small.

Thanks,
Kevin
One other thing to add. I have concidered using strongly typed dataset (in memory), but since I am not familiar with the numerous collection classes in .NET, I decided to ask in this forum in hopes of sparking up some dialog on any pros and cons of possible solutions.

Thanks,
Kevin
DataSet IS the object you will want to work with if you choose to do it solely in .NET. Unless you wanted to create your own custom class to do the work for you, that would always be fun. I would imagine you could set up some sort textreader as the datasource for the dataset and have it automatically fill from the different files.