# Need Help with Data File Manipulation



## Guest (Sep 12, 2007)

I am using x,y,z data sets that consist of ~ 1E06 data points for my graduate project. I am able to generate the data set from a NOAA Mapping website which they send to me as a .txt file. Typically I would just use the Excel Import Wizard to create a delimited file based on spaces, tabs, commas, etc. and then save as a .dat file to import into the model. In this case, the data is so voluminous that it exceeds the 65,536 row limitation and ends up truncating the import.

Ideally, I need to create a file so I can spatially sort the data and create transects. Those transects would certainly have &lt;&lt; 65,536 rows (data points) but I can't quite seem to get there since the file prematurely truncates.

Any ideas for how I can overcome the row size limitation?

Thanks.

JR


----------



## udpolo15 (Sep 12, 2007)

Access might be the easiest.


----------



## Guest (Sep 12, 2007)

^^^ Thanks!! I looks as if I was able to import all of the data. lusone: Now I just have to learn how to manipulate MS Access - I never had much use for it.

JR


----------



## IlPadrino (Sep 13, 2007)

Let me know if you need help with Access... I haven't played with it much since my thesis in 2004, but at the time I did a lot with XML and databases. Me and Access got to know one another quite well!

XML would have been another way of solving this problem, but the transformations can get messy.

Scott-


----------



## MA_PE (Sep 13, 2007)

The power of BASIC for this type of operation cannot be underestimated. My recommendation is to write (or have someone write) a simple program to do the mainipulation. Visual Basic/ older DOS-based BASIC/ MatLAB/ etc. will do this pretty effortlessly.

JR I'm sure you've had some programming along the way. :happy:


----------



## Guest (Sep 13, 2007)

IlPadrino said:


> Let me know if you need help with Access...


Thanks! 



MA_PE said:


> The power of BASIC for this type of operation cannot be underestimated. My recommendation is to write (or have someone write) a simple program to do the mainipulation. Visual Basic/ older DOS-based BASIC/ MatLAB/ etc. will do this pretty effortlessly.
> JR I'm sure you've had some programming along the way. :happy:


Actually my professor and I discussed this very approach today - he has typically written scripts in MatLAB to pre-sort and smooth the data. Most of the modeling work I have done has been easily accomplished through creating my own .dat files through Excel - I have never had to employ any programming language much less actually write a macro or subroutine to import the data. Even though I have taken C++ and Visual Basic in college - it has been many years and I typically hated the programming end of problem solving.

My professor has a student license to offer for some sophisticated pre-processors - I am hoping to be able to utilize tools within those programs to obtain what I need. lease: Also, don't forget - I am on a schedule to graduate this semester. I need to get through this with as little pain as possible.

JR


----------



## IlPadrino (Sep 13, 2007)

jregieng said:


> Also, don't forget - I am on a schedule to graduate this semester. I need to get through this with as little pain as possible.


Amen! No extra credit is given for working harder. Git 'er done!


----------



## Guest (Sep 21, 2007)

*[SIZE=18pt]*** UPDATE ***[/SIZE]*

I am using a software package called Suface Water Modeling System (SMS) version 9.0 produced by Environmental Modeling Systems, Inc. (EMS-i). Very nice package but difficult to navigate if you are short on time.

I was finally able to achieve extracting a cross-section from manipulated data and export that cross-section into a USEFUL file that can be used, in part, as data input for my 1-D model.

This weekend will be spent developing my case utilizing one (1) transect and then I can roll out the analysis for the remainder of the transects and scenarios necessary and finally call this bitch done! :woot: I can't finish this fast enought :true:

:bio:

JR


----------

