Skip to content

Conversation

@ldecicco-USGS
Copy link
Collaborator

No description provided.

Copy link
Collaborator

@ehinman ehinman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking good. I want to do some actual testing of the functions on Friday, but I've taken a quick look at the diffs. A few small comments.

# expect_equal(chi_iv$dateTime[nrow(chi_iv)], as.POSIXct("2014-05-01T12:00",
# format = "%Y-%m-%dT%H:%M",
# tz = "America/Chicago"
# ))
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this just to cut down on testing time, especially since we have a replacement continuous function now?

grepl("/", datetime)){
return(datetime)
} else {
datetime1 <- tryCatch({
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are you catching here? The following works without warning for me:

lubridate::as_datetime("2010-01-01T00:00:15Z")

Are you catching something else?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So this is funky:

> lubridate::as_datetime("2010-01-01T00:00Z")
[1] "2020-10-01 01:00:00 UTC"
> lubridate::as_datetime("2014-05-01T00:00Z")
[1] NA
Warning message:
All formats failed to parse. No formats found. 

# Move other id columns:
return_list <- move_id_col(return_list,
output_id)
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice that you moved this out of the individual functions, plus the column moving piece.

Co-authored-by: Elise Hinman <121896266+ehinman@users.noreply.github.com>
@ehinman
Copy link
Collaborator

ehinman commented Jan 2, 2026

General comment: I would change the no_paging documentation to say something like "This can be dangerous because the number of rows returned will cut off at the limit value (default maximum limit is currently 50,000 rows) without indication that more data are available. Use TRUE with caution."

@ehinman
Copy link
Collaborator

ehinman commented Jan 2, 2026

Interesting that the service does not stop you from trying to download a date range that goes back in time:

test <- read_waterdata_continuous(
  monitoring_location_id = "USGS-15300500",
  parameter_code = "00065",
  time = "2023-10-02T00:00:00Z/2023-10-01T21:00:00Z", # can also do c("2023-10-02T00:00:00Z", "2023-10-01T21:00:00Z")
  no_paging = TRUE
)

@ehinman
Copy link
Collaborator

ehinman commented Jan 2, 2026

This is a pretty edge case, but when I ran the following without my API key, I got this error:

test <- read_waterdata_continuous(
   monitoring_location_id = "USGS-15300500",
   time = "../2023-10-01T21:00:00Z"
)

Requesting:
https://api.waterdata.usgs.gov/ogcapi/v0/collections/continuous/items?f=json&lang=en-US&skipGeometry=TRUE&monitoring_location_id=USGS-15300500&time=..%2F2023-10-01T21%3A00%3A00Z&limit=50000
Error in walk_pages(req) : HTTP 429 Too Many Requests.

When I try it with no_paging = TRUE, I get a little more info on what I need to do:

test <- read_waterdata_continuous(
   monitoring_location_id = "USGS-15300500",
   time = "../2023-10-01T21:00:00Z",
   no_paging = TRUE
)
Requesting:
https://api.waterdata.usgs.gov/ogcapi/v0/collections/continuous/items?f=csv&lang=en-US&skipGeometry=TRUE&monitoring_location_id=USGS-15300500&time=..%2F2023-10-01T21%3A00%3A00Z&limit=50000
Error in `httr2::req_perform()` at dataRetrieval/R/walk_pages.R:110:3:
! HTTP 429 Too Many Requests.
ℹ You have exceeded your rate limit. Make sure you provided your API key
  from https://api.waterdata.usgs.gov/signup/, then either try again later
  or contact us at
  https://waterdata.usgs.gov/questions-comments/?referrerUrl=https://api.waterdata.usgs.gov
  for assistance.

Possible to make the behavior the same across scenarios?

Copy link
Collaborator

@ehinman ehinman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This all looks good to me, and my tests worked fine. I'm approving this PR, though I have some optional comments to consider, either already left above, or see below:

I'm having a hard time finding where attributes are documented for the water data API functions. I have a feeling most people using the functions won't even know about the available attributes (I had to Google how to see all the attributes, myself 😆 ). It might be nice to document this somewhere in the Water Data API documentation/vignettes (I do see some documentation using the older functions).

@ldecicco-USGS
Copy link
Collaborator Author

This is a pretty edge case, but when I ran the following without my API key, I got this error:

test <- read_waterdata_continuous(
   monitoring_location_id = "USGS-15300500",
   time = "../2023-10-01T21:00:00Z"
)

Requesting:
https://api.waterdata.usgs.gov/ogcapi/v0/collections/continuous/items?f=json&lang=en-US&skipGeometry=TRUE&monitoring_location_id=USGS-15300500&time=..%2F2023-10-01T21%3A00%3A00Z&limit=50000
Error in walk_pages(req) : HTTP 429 Too Many Requests.

When I try it with no_paging = TRUE, I get a little more info on what I need to do:

test <- read_waterdata_continuous(
   monitoring_location_id = "USGS-15300500",
   time = "../2023-10-01T21:00:00Z",
   no_paging = TRUE
)
Requesting:
https://api.waterdata.usgs.gov/ogcapi/v0/collections/continuous/items?f=csv&lang=en-US&skipGeometry=TRUE&monitoring_location_id=USGS-15300500&time=..%2F2023-10-01T21%3A00%3A00Z&limit=50000
Error in `httr2::req_perform()` at dataRetrieval/R/walk_pages.R:110:3:
! HTTP 429 Too Many Requests.
ℹ You have exceeded your rate limit. Make sure you provided your API key
  from https://api.waterdata.usgs.gov/signup/, then either try again later
  or contact us at
  https://waterdata.usgs.gov/questions-comments/?referrerUrl=https://api.waterdata.usgs.gov
  for assistance.

Possible to make the behavior the same across scenarios?

I'll make an issue for this.

@ldecicco-USGS ldecicco-USGS merged commit 872d4fe into DOI-USGS:develop Jan 2, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants