Automating water quality assessments

The Utah DWQ experience

Jake Vander Laan ()

IR Team: Elise Hinman & Emilie Flemer

To view slides:

bookdown.org/jakevl/udwq-r-tools/

(May be a bit slow loading)

Challenges



Philosophical challenges

We all want assessments that are:

  • Comprehensive
  • Inclusive
  • Accurate
  • Transparent
  • Certain
  • Efficient

Philosophical challenges

  • Data richness vs. monitoring resources
  • Comprehensiveness vs. speed
  • Date inclusivity vs. data assembly
  • Assessment certainty vs. assessment efficiency
  • Automation vs. professional judgment
  • Black-box vs. transparent

Philosophical challenges


Philosophy is for managers…

Staff tend to focus on the technical challenges.

Technical challenges

Integrating data, standards, methods, & results

Data: too much & not enough


  • 855 assessment units
  • 622 use geographic use assignments
  • >1000 unique criteria
    • Site specific
    • Temoprally specific

Harmonizing & integrating multiple datasets

Incorporate multiple data sources:

  1. Internal data
  2. Partner agencies
  3. Stakeholders

Interpretation and analysis issues:
1. Field & lab methods
2. Parameters
3. Site types

Batching assessments

Previous approaches:

  • Site by site
  • Parameter by paramter
  • Criterion by criterion


Digitizing criteria

Utah water quality standards

Criteria must be digitized along with temporal & spatial metadata to link criteria and data automatically.

Text format


Flattened table


Temporal & spatial compenents

Other issues

  • Documentation
  • Communicating results
  • Treadmill timeline
  • Technical deficit

Opportunities

  1. Data in one usable format, connected to criteria and spatial and temporal metadata
  2. Significant resources focused on a uniform set of issues
  3. Build credibility and trust
  4. Multipurpose tools
  5. Understanding higher level information

Utah DWQ’s approach

Motivation and objectives

Motivation

Rapid, repeatable, and adaptable assessment process

Objectives

1. Flexible tool inputs
2. Re-usable, generalizable, standardized components
3. Transparent & documented process
4. Readily reviewable

Toolsets

Speaking of technical deficit…


R

R is an open source statistical program and coding language.
r-project.org

Robust user community of coders & problem solvers.
Fully customizable.

Everything that exists is an object.
Everything that happens is a function call.
— John Chambers

Functions & packages
Function
A block of code that performs a particular task or set of tasks. A function has a name, arguments, a body (code), and a return value.

Argument
Change-able inputs to a function.

Package
A portable collection of R functions.

R documentation

GitHub

Collaborative code writing, versioning, & sharing platform.
Individuals, organizations, teams, and repositories.

DWQ GitHub organization
A location for DWQ staff to store, collaborate, and share repositories.
github.com/utah-dwq

DWQ R packages
DWQ has two packages in current development. These can be viewed and installed via GitHub.

Installing from GitHub

install.packages('devtools')
devtools::install_github('utah-dwq/irTools')
library(irTools)
asmntDashboard()

Data portals & web services

Water quality data portal
waterqualitydata.us/portal

ECHO
Web service capabilities 

library(wqTools)
two_site_nr=readWQP(type="narrowresult",
                    siteid=c("UTAHDWQ_WQX-4900440","UTAHDWQ_WQX-4900470"), 
                    start_date="01/01/2016", end_date="12/31/2018")

utah_lake_nr=readWQP(type="narrowresult",
                     auid=c('UT-L-16020201-004_01', 'UT-L-16020201-004_02'),
                     start_date="01/01/2016", end_date="12/31/2018",
                     siteType=c("Lake, Reservoir, Impoundment","Stream"),
                     print=F)


R-Shiny

Shiny is an R package for building interactive web applications to visualize and analyze data and communicate analytical results. These can be run locally, hosted on the web, and embedded into documents, presentations, and web pages.

Assessment process


Assembling data

Water Quality Portal 

library(wqTools)
downloadWQP(outfile_path='C:/Your/Folder/Path', 
            start_date="01/01/2018", end_date="12/31/2018", 
            retrieve=c("narrowresult","sites","detquantlim","activity"))

Other datasets

  • High frequency DO
  • Macroinvertebrates
  • Phytoplankton

Screening data

  • Table-based validation
  • Tables are unique-d subsets of metadata
  • Reviews accumulate cycle-to-cycle
  • Functions to update & apply data validation tables
  • Results in screened data with consistent nomenclature



Screening data

  1. Labs & activity types
  2. Activity media
  3. Parameter names & fractions
  4. Detection conditions
  5. Detection limit types
  6. Unit conversion factors
  7. Apply screening tables and subset to accepted data

Screening data

Assigning criteria

assignCriteria(data, crit_wb, crit_sheetname, ss_sheetname,
               crit_startRow = 1, ss_startRow = 1, rm_nocrit = TRUE)

Formula derived criteria

Hardness, temperature, & pH dependent criteria


Formula derived criteria

Assessment methods as functions

Methods




Functions

Count exceedances


Assess counts

Review tools

Interactive tools for visualizing sites, data, and assessment results and providing reviewer feedback.


Demonstrations

Site validation

library(irTools)
siteValApp()



Assessment dashboard

library(irTools)
asmntDashboard()





Lake dashboard

udwq.shinyapps.io/lakedashboard


Utah Lake Data Explorer

udwq.shinyapps.io/UtahLakeDataExplorer


High frequency DO

udwq.shinyapps.io/hfdoDashboard


Questions