Wintertime Posted May 8, 2008 Share Posted May 8, 2008 According to the NGS website, the datasheet archives have not been updated since October 2006. I thought there was a place to get month-by-month updates to cover the changes made in a given county since the last archiving process was performed, but I can't find it. So how do I get up-do-date .dat files for an entire county? Patty Quote Link to comment
+ltleelim Posted May 8, 2008 Share Posted May 8, 2008 According to the NGS website, the datasheet archives have not been updated since October 2006. I thought there was a place to get month-by-month updates to cover the changes made in a given county since the last archiving process was performed, but I can't find it. So how do I get up-do-date .dat files for an entire county? I've never used the archives because they're always out of date. I'm not sure what other people do, but when I need a whole county then I use the County retrieval method to retrieve the latest datasheets for the county. If you select too many stations, the request will take too long and timeout. You have to select subsets of the list and paste the results together afterwards. I prefer to sort the list by PID to make it easier to figure out where the next request should start. It's not too hard and the datasheets are always up to date. Lloyd Quote Link to comment
Wintertime Posted May 8, 2008 Author Share Posted May 8, 2008 I'm not sure what other people do, but when I need a whole county then I use the County retrieval method to retrieve the latest datasheets for the county. I thought about that, but wasn't sure whether (1) I could download a whole county without something breaking (your suggestion of doing it in batches is a good idea); and (2) the resulting file would work with the DAT2KML awk script. But I just looked at some old .dat and .txt files I have from the NGS database, and they appear to be identical. So maybe I can just change the extension on the file, and DAT2KML will be happy. Anyone tried that? Patty Quote Link to comment
+Klemmer Posted May 9, 2008 Share Posted May 9, 2008 (edited) Patty: No I haven't tried it, but your question just beat me to it. I was going to raise the same question. I have a bigger problem: the counties down here are so large, to get the whole county, I would have to do it in many segments. It will be a real pain. For example, my home county (Orange) is aver 2800 marks, and I get around to most of the areas in the county fairly often. LA county is over 8000 marks...... NGS folks: So - I sure would like to get some full county .dat files again. There are 6 counties in SoCal that I use from several times per year up to several times per month (some more than others, of course). Can anyone at NGS help me out with this? Counties I would like: San Diego, Los Angeles, Orange, San Bernardino, Imperial, Riverside. Klemmer Edit: Disregard my comment on Yosemite area if you saw it. My mistake. Above still an issue. Edited May 9, 2008 by Klemmer & TeddyBearMama Quote Link to comment
Wintertime Posted May 9, 2008 Author Share Posted May 9, 2008 Patty:No I haven't tried it, but your question just beat me to it. I was going to raise the same question. I went ahead and tried it, and it worked fine. Using Lloyd's tip, to avoid problems, I downloaded the Tuolumne County datasheets in six segments, then used a text editor to combine them. I don't know how many total marks there are, or what the largest segment was, but it worked fine. I copied the file and renamed the copy as .dat instead of .txt; DAT2KML converted it just fine. Still, it would be nice to have updated .dat files on the NGS site. Patty: P.S. I was able to get the Yosemite area counties we will need in one piece. Cool! I've already done Tuolumne, but will talk to you off-line about getting the others. Patty Quote Link to comment
foxtrot_xray Posted May 9, 2008 Share Posted May 9, 2008 Since I'm data-scraping, I'd like to second this. I noticed that a few SE stated had their state archived updated earlier this year (as opposed to the others in '06), including my home state. (Which I appreciate!) Would like to see all the other states updated too. Quote Link to comment
holograph Posted May 10, 2008 Share Posted May 10, 2008 If you have any way to handle shapefiles, you can download the shapefiles by state. The shapefiles have been kept up to date for the most part. It looks like all the states were done around Dec 2007, and a few states have been updated more recently than that. The state datasheet archives, and the monthly datasheet update archives haven't been updated since 2006. Quote Link to comment
Wintertime Posted May 10, 2008 Author Share Posted May 10, 2008 Using Lloyd's tip, to avoid problems, I downloaded the Tuolumne County datasheets in six segments, then used a text editor to combine them. Update: Larry and I have been downloading some other Yosemite-area county files all at once. I.e., select a county, get the listing of stations, select all, and get datasheets for all of them. So far, the only problem either of us has encountered was when he tried to get a county that had more than 1,000 marks. He had to do that in two pieces. But the NGS site has behaved fine for us so far even on counties with several hundred datasheets. So far, no timeouts as Lloyd encountered. Perhaps we're doing it at a time of day when the professional "benchmark hunters" (i.e., surveyors) have all gone home for the day. Patty Quote Link to comment
foxtrot_xray Posted May 12, 2008 Share Posted May 12, 2008 If you have any way to handle shapefiles, you can download the shapefiles by state. The shapefiles have been kept up to date for the most part. It looks like all the states were done around Dec 2007, and a few states have been updated more recently than that. Yes - but keep in mind that some of us need more than just the shape files. (For example, if all I needed was GPS coords, then I'd Geocache, and not benchmark hunt! ) And some of the SE states HAVE had archives since 2006, as well, like I had previously said. Update: Larry and I have been downloading some other Yosemite-area county files all at once. I.e., select a county, get the listing of stations, select all, and get datasheets for all of them. So far, the only problem either of us has encountered was when he tried to get a county that had more than 1,000 marks. He had to do that in two pieces. But the NGS site has behaved fine for us so far even on counties with several hundred datasheets. So far, no timeouts as Lloyd encountered. Perhaps we're doing it at a time of day when the professional "benchmark hunters" (i.e., surveyors) have all gone home for the day. If it's possible, could either of you (or anyone else that pulls them) send them to me, I'd love to be able to update some of the GE database. Quote Link to comment
holograph Posted May 12, 2008 Share Posted May 12, 2008 You are welcome to download the archives that I've acquired by going to http://www.holoscenes.com/special/geocac/ds_archive.html These will not be exactly the same as a snapshot of the current NGS datasheets, because the way they were built was : I obtained the NGS archives in 2006 when they became available. Each month since, I've downloaded all the datasheets updated in that month. I've replaced or added the original downloaded datasheets with the updates. This means, for instance, that datasheets that haven't been updated since 2006 won't have the current NAD83(2007) datum and accuracy estimates. For most purposes that probably won't matter. Legal disclaimer: Use at your own risk! I take no responsibility for the completeness or accuracy of the archives. It was a fairly cumbersome process to generate the 3,219 county archive files and upload them to my web site, so I probably won't update these each month. Quote Link to comment
foxtrot_xray Posted May 14, 2008 Share Posted May 14, 2008 You are welcome to download the archives that I've acquired by going to http://www.holoscenes.com/special/geocac/ds_archive.htmlThanks! It'll get my GE database up to snuff, then. (And yeah, for myself, anywho, the unupdated marks won't matter.) Is there a function on the NGS site that I missed - that allows you to download only updated sheets? Cheers! Mike. Quote Link to comment
holograph Posted May 14, 2008 Share Posted May 14, 2008 (edited) You can go to http://www.ngs.noaa.gov/cgi-bin/ds_dates.prl to retrieve datasheets by load date. The retrieval mechanism is the same as the other lookups -- you get a list of PIDs, and you select from them to retrieve the datasheets. By putting in a date range beginning with the last time you retrieved the updates, you get all new updates. edit: P.S. if you are going to download all 3200+ counties, then please pace the downloads so that the site isn't overloaded. The hosting service will probably send me a nasty-gram if the site consumes their server. Edited May 14, 2008 by holograph Quote Link to comment
foxtrot_xray Posted May 14, 2008 Share Posted May 14, 2008 You can go to http://www.ngs.noaa.gov/cgi-bin/ds_dates.prl to retrieve datasheets by load date. The retrieval mechanism is the same as the other lookups -- you get a list of PIDs, and you select from them to retrieve the datasheets. By putting in a date range beginning with the last time you retrieved the updates, you get all new updates. Well, I'll be! I took 'load date' as pretty much the 'set date' of the monument - meaning it would have only given me stations that were MONUMENTED after the date I entered. Al-righty then, thanks. edit: P.S. if you are going to download all 3200+ counties, then please pace the downloads so that the site isn't overloaded. The hosting service will probably send me a nasty-gram if the site consumes their server. Oh, crap, sorry. I'm stopping now. If you get in trouble or get fined or whatever, please let me know. I'll make it up to you. PM or E-mail me. Thanks! Mike. Quote Link to comment
holograph Posted May 15, 2008 Share Posted May 15, 2008 Don't worry about it. The storage and bandwidth are well within the site's limits. It's just that if you have a 'bot or program that is downloading multiple files at a time, or is running non-stop for an hour, it will probably slow down access to the site. And since the site shares a server with other sites, I don't know what other sites are affected. If you want to download all the files, please just do them one at a time, and take a few seconds break between files. Quote Link to comment
+Klemmer Posted May 17, 2008 Share Posted May 17, 2008 (edited) Thanks, Holograph! That was painless, and now I'm updated in all of SoCal for the first time since Oct06. Appreciate your hard work. Edited May 17, 2008 by Klemmer & TeddyBearMama Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.