Add example of converting user data to EGRET format
woodwards opened this issue · comments
From the documentation I can only see examples using existing NWIS files or existing Daily and INFO dataframes.
Many users like myself will have data in different file formats or in dataframes in their session. Can you provide some guidance on how to convert such data into the right format for Daily and INFO? I can't find any information on the required columns in these dataframes.
Can you provide a vignette that constructs minimal Daily and INFO from scratch?
Simon: the way this is done is all laid out in the vignette that comes with the EGRET package. I've attached it here for your convenience. The part that would be of interest is on pages 7-15. The specific functions for these purposes are readUserSample, readUserDaily, and readUserInfo.
Let us know if that doesn't meet your needs.
Robert Hirsch
***********************************************************
Robert M. Hirsch, Research Hydrologist Emeritus
430 National Center
U.S. Geological Survey rhirsch@usgs.gov
Reston, VA 20192 703-648-5888
https://www.usgs.gov/staff-profiles/robert-hirsch
EGRET software http://usgs-r.github.io/EGRET/
***********************************************************
…________________________________
From: Simon Woodward <notifications@github.com>
Sent: Saturday, February 1, 2020 2:37 PM
To: USGS-R/EGRET <EGRET@noreply.github.com>
Cc: Subscribed <subscribed@noreply.github.com>
Subject: [EXTERNAL] [USGS-R/EGRET] Add example of converting user data to EGRET format (#249)
From the documentation I can only see examples using existing NWIS files or existing Daily and INFO dataframes.
Many users like myself will have data in different file formats or in dataframes in their session. Can you provide some guidance on how to convert such data into the right format for Daily and INFO? I can't find any information on the required columns in these dataframes.
Can you provide a vignette that constructs Daily and INFO from scratch?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub<#249>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAI53UJ4NTTUK4EDO46TA2TRAXFRLANCNFSM4KOVACXQ>.
Thank you Robert.
Ok I see the instructions now. In my case the data is in a dataframe in the R environment, rather than in a file. I suppose I could write it to a text file and then read it back in. I have two solutes.
> glimpse(tdata)
Observations: 5,479
Variables: 9
$ shortname <chr> "Ta", "Ta", "Ta", "Ta", "Ta", "Ta", "Ta", "Ta", "Ta", "Ta", "Ta", "Ta", ...
$ TF <dbl> 4.54200, 4.36800, 4.19800, 4.36667, 4.40000, 4.03000, 4.06000, 3.84500, ...
$ TP <dbl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, ...
$ TN <dbl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, ...
$ x <int> 366, 367, 368, 369, 370, 371, 372, 373, 374, 375, 376, 377, 378, 379, 38...
$ xdate <date> 2002-01-01, 2002-01-02, 2002-01-03, 2002-01-04, 2002-01-05, 2002-01-06,...
$ day <int> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 2...
$ month <dbl> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
$ year <dbl> 2002, 2002, 2002, 2002, 2002, 2002, 2002, 2002, 2002, 2002, 2002, 2002, ...
>
It occurred to me after I wrote this that the INFO and Daily dataframes might be described in the written documentation. Thanks for your help.
The readDataFromFile function doesn't seem to work. It doesn't normalise the column names or add the LogQ column, and it returns Q as chr. Also it doesn't add the JulianMonthDayDecYearMonthSeq expected by mergeReport.
# convert my data to EGRET format
Daily <- data.frame(Date=tdata$xdate, Q=tdata$TF, LogQ=log(tdata$TF))
write_csv(Daily, "temp_egret_daily.csv")
Daily <- readDataFromFile(getwd(), "temp_egret_daily.csv")
Daily$Q <- as.numeric(Daily$Q)
Sample <- data.frame(Date=tdata$xdate, Remark="", TP=tdata$TP) %>% drop_na()
write_csv(Sample, "temp_egret_sample.csv")
Sample <- readUserSample(getwd(), "temp_egret_sample.csv")
INFO <- data.frame(param.units="mg/L",
shortName=tdata$shortname[1],
paramShortName="TP",
drainSqKm=1,
constitAbbrev="TP",
staAbbrev=tdata$shortname[1],
paStart=1,
paLong=60,
queryTime=Sys.time()
stringsAsFactors=FALSE)
write_csv(INFO, "temp_egret_info.csv")
INFO <- readUserInfo(getwd(), "temp_egret_info.csv", interactive=FALSE)
edata <- mergeReport(INFO, Daily, Sample)
Discharge Record is 5479 days long, which is 15 years
First day of the discharge record is 2002-01-01 and last day is 2016-12-31
The water quality record has 180 samples
The first sample is from 2002-01-30 and the last sample is from 2016-12-15
Discharge: Minimum, mean and maximum 1.82 4.37 52.4
Concentration: Minimum, mean and maximum 0.032 0.066 0.79
Percentage of the sample values that are censored is 0 %
Daily data frame expecting columns: JulianMonthDayDecYearMonthSeq
There's a mistake in the user manual page 9
Daily <-readDataFromFile(filePath,fileName,
separator="\t")
should say
Daily <-readUserDaily(filePath,fileName,
separator="\t")
Is there any user forum for this software? I have a lot of questions.
I assume you have worked it out, but just in case. When I have situations like this (Sample data in an R data frame) I do just turn it into a 3 column csv (Date, remark, and value) and then read in the csv using readUserSample. Regarding your other comment about readDataFromFile, I wasn't really aware that we had such a function. To me it always seems best to just use the three specific ones: readUserSample, readUserDaily, and readUserInfo. They are pretty straightforward.
Bob Hirsch
***********************************************************
Robert M. Hirsch, Research Hydrologist Emeritus
430 National Center
U.S. Geological Survey rhirsch@usgs.gov
Reston, VA 20192 703-648-5888
https://www.usgs.gov/staff-profiles/robert-hirsch
EGRET software http://usgs-r.github.io/EGRET/
***********************************************************
________________________________
From: Simon Woodward <notifications@github.com>
Sent: Sunday, February 2, 2020 1:47 PM
To: USGS-R/EGRET <EGRET@noreply.github.com>
Cc: Hirsch, Robert M <rhirsch@usgs.gov>; Comment <comment@noreply.github.com>
Subject: [EXTERNAL] Re: [USGS-R/EGRET] Add example of converting user data to EGRET format (#249)
Thank you Robert.
Ok I see the instructions now. In my case the data is in a dataframe in the R environment, rather than in a file. I suppose I could write it to a text file and then read it back in. I have two solutes.
glimpse(tdata)
Observations: 5,479
Variables: 9
$ shortname <chr> "Ta", "Ta", "Ta", "Ta", "Ta", "Ta", "Ta", "Ta", "Ta", "Ta", "Ta", "Ta", ...
$ TF <dbl> 4.54200, 4.36800, 4.19800, 4.36667, 4.40000, 4.03000, 4.06000, 3.84500, ...
$ TP <dbl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, ...
$ TN <dbl> NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, NA, ...
$ x <int> 366, 367, 368, 369, 370, 371, 372, 373, 374, 375, 376, 377, 378, 379, 38...
$ xdate <date> 2002-01-01, 2002-01-02, 2002-01-03, 2002-01-04, 2002-01-05, 2002-01-06,...
$ day <int> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 2...
$ month <dbl> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
$ year <dbl> 2002, 2002, 2002, 2002, 2002, 2002, 2002, 2002, 2002, 2002, 2002, 2002, ...
It occurred to me after I wrote this that the INFO and Daily dataframes might be described in the written documentation. Thanks for your help.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub<#249>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAI53UMJHXDV4IGALRXDXQDRA4IK5ANCNFSM4KOVACXQ>.
Thanks Robert
And just to mention again that the documentation incorrectly uses readDataFromFile on page 9
I have got it working now.
Simon