Need Help with Data File Manipulation

[Guest]

I am using x,y,z data sets that consist of ~ 1E06 data points for my graduate project. I am able to generate the data set from a NOAA Mapping website which they send to me as a .txt file. Typically I would just use the Excel Import Wizard to create a delimited file based on spaces, tabs, commas, etc. and then save as a .dat file to import into the model. In this case, the data is so voluminous that it exceeds the 65,536 row limitation and ends up truncating the import.

Ideally, I need to create a file so I can spatially sort the data and create transects. Those transects would certainly have << 65,536 rows (data points) but I can't quite seem to get there since the file prematurely truncates.

Any ideas for how I can overcome the row size limitation?

Thanks.

JR

 

[udpolo15]

Access might be the easiest.

 

[Guest]

^^^ Thanks!! I looks as if I was able to import all of the data. lusone: Now I just have to learn how to manipulate MS Access - I never had much use for it.

JR

 

[IlPadrino]

Let me know if you need help with Access... I haven't played with it much since my thesis in 2004, but at the time I did a lot with XML and databases. Me and Access got to know one another quite well!

XML would have been another way of solving this problem, but the transformations can get messy.

Scott-

 

[MA_PE]

The power of BASIC for this type of operation cannot be underestimated. My recommendation is to write (or have someone write) a simple program to do the mainipulation. Visual Basic/ older DOS-based BASIC/ MatLAB/ etc. will do this pretty effortlessly.

JR I'm sure you've had some programming along the way. :happy:

 

[Guest]

Let me know if you need help with Access...

Thanks!

The power of BASIC for this type of operation cannot be underestimated. My recommendation is to write (or have someone write) a simple program to do the mainipulation. Visual Basic/ older DOS-based BASIC/ MatLAB/ etc. will do this pretty effortlessly.
JR I'm sure you've had some programming along the way. :happy:

Actually my professor and I discussed this very approach today - he has typically written scripts in MatLAB to pre-sort and smooth the data. Most of the modeling work I have done has been easily accomplished through creating my own .dat files through Excel - I have never had to employ any programming language much less actually write a macro or subroutine to import the data. Even though I have taken C++ and Visual Basic in college - it has been many years and I typically hated the programming end of problem solving.

My professor has a student license to offer for some sophisticated pre-processors - I am hoping to be able to utilize tools within those programs to obtain what I need. lease: Also, don't forget - I am on a schedule to graduate this semester. I need to get through this with as little pain as possible.

JR

 

[IlPadrino]

Also, don't forget - I am on a schedule to graduate this semester. I need to get through this with as little pain as possible.

Amen! No extra credit is given for working harder. Git 'er done!

 

[Guest]

[SIZE=18pt]*** UPDATE ***[/SIZE]

I am using a software package called Suface Water Modeling System (SMS) version 9.0 produced by Environmental Modeling Systems, Inc. (EMS-i). Very nice package but difficult to navigate if you are short on time.

I was finally able to achieve extracting a cross-section from manipulated data and export that cross-section into a USEFUL file that can be used, in part, as data input for my 1-D model.

This weekend will be spent developing my case utilizing one (1) transect and then I can roll out the analysis for the remainder of the transects and scenarios necessary and finally call this ***** done! :woot: I can't finish this fast enought :true:

:bio:

JR