Repository navigation
Need feedback from Product on how to handle real world csv errors #522
Copy link
Copy link
Open
Description
I need guidance on how to handle all of the issues in the well-inventory-template.
questions
- should we assume 0 if mp_height is blank?
- should we auto calculate elevation if elevation_ft is blank?
- should we add "Monitoring Complete" to "monitoring_frequency" lexicon category?
- should rows with blank well_name_point_Id trigger an autogenerated name? what prefix should be used by default NM-?
- how should we handle blank depth_to_water_ft? Should it be nullable or 0, etc?
- should rows without issues be imported or does the entire CSV need to be correct before any wells are imported?
@marissafichera @Rachel-Hobbs @savenoch
https://docs.google.com/spreadsheets/d/13_g0dSK4iHBAEGYYT5Af3mQbW0S0Sj_WAh0r1PxkMqk/edit?usp=sharing
oco well-inventory-csv tests/features/data/well-inventory-real-user-entered-data.csv
[WELL INVENTORY IMPORT] COMPLETED WITH ISSUES
========================================================================
SUMMARY
---------------------------
processed | 129
imported | 0
rows_with_issues | 129
VALIDATION
Validation errors: 607
Most common validation errors:
# | field | count | error
------------------------------------------------------------------------------------------------------------------------------------------------
1 | mp_height | 112 | Input should be a valid number, unable to parse string as a number
2 | elevation_method | 103 | Input should be 'Altimeter', 'Differentially corrected GPS', 'Survey-grade GPS', 'Global...
3 | elevation_ft | 99 | Input should be a valid number, unable to parse string as a number
4 | monitoring_frequency | 82 | Input should be 'Monthly', 'Bimonthly', 'Bimonthly reported', 'Quarterly', 'Biannual', 'Annual',...
5 | depth_to_water_ft | 68 | Input should be a valid number, unable to parse string as a number
Row 1 (8 issues)
1. utm_easting: Input should be a valid number, unable to parse string as a number
input:
2. utm_northing: Input should be a valid number, unable to parse string as a number
input:
3. elevation_ft: Input should be a valid number, unable to parse string as a number
input:
4. elevation_method: Input should be 'Altimeter', 'Differentially corrected GPS', 'Survey-grade GPS', 'Global positioning system (GPS)', 'LiDAR DEM', 'Level or other survey method', 'Interpolated
from topographic map', 'Interpolated from digital elevation model (DEM)', 'Reported', 'Survey-grade Global Navigation Satellite Sys, Lvl1', 'USGS National Elevation Dataset
(NED)' or 'Unknown'
input:
5. measuring_point_height_ft: Input should be a valid number, unable to parse string as a number
input:
6. monitoring_frequency: Input should be 'Monthly', 'Bimonthly', 'Bimonthly reported', 'Quarterly', 'Biannual', 'Annual', 'Decadal' or 'Event-based'
input: Monitoring complete
7. mp_height: Input should be a valid number, unable to parse string as a number
input:
8. depth_to_water_ft: Input should be a valid number, unable to parse string as a number
input:
--------------------------------------------------------
Row 2 (4 issues)
1. elevation_ft: Input should be a valid number, unable to parse string as a number
input:
2. elevation_method: Input should be 'Altimeter', 'Differentially corrected GPS', 'Survey-grade GPS', 'Global positioning system (GPS)', 'LiDAR DEM', 'Level or other survey method', 'Interpolated
from topographic map', 'Interpolated from digital elevation model (DEM)', 'Reported', 'Survey-grade Global Navigation Satellite Sys, Lvl1', 'USGS National Elevation Dataset
(NED)' or 'Unknown'
input:
... and 597 more validation errors
========================================================================
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels
Activity
github-actions commentedon Feb 15, 2026
Linked Jira ticket created: BDMS-560
ksmuczynski commentedon Feb 16, 2026
@jirhiker I’ve had similar questions while running import tests on the AMP well inventory data. My thoughts are below.
I would defer to AMMP, but in my opinion, yes.
Again, I'd defer to AMMP, but in my opinion, yes.
No. In my opinion,
Monitoring Completeis more indicative of a Monitoring Status than a Monitoring Frequency and should be mapped as follows:StatusHistory.status_type=Monitoring Statusand StatusHistory.status_value =Not currently monitored. Alternatively, we could addMonitoring Completeto thestatus_valuelexicon, but I would first ask AMMP to clarify the meaning ofMonitoring Completeand see how/if it differs fromNot currently monitored. For additional context, the definition ofNot currently monitoredis a wellnot currently being monitored by AMMP.I'm not sure how I feel about auto-generated names, especially if we decide that complete records can be imported despite errors still existing in the csv import file. In terms of naming conventions, I was recently thinking that county abbreviations might be a good choice moving forward.
It should be nullable with a flag/qualifier. Currently, the
Observationmodel usesgroundwater_level_reasonto provide context for a NULL value. Side note, I would like to generalize this at a later time tofield_qualifierand add anotherlab_qualifierfield.Yes, I think rows without issues should be imported. The entire CSV does not need to be correct before data is imported. It would also be useful to have a log/message at the end noting the number of records imported out of total number of records (e.g. 12 records imported out of 20 records). A separate, additional error log would also help point users in the right direction to fix any issues.
jirhiker commentedon Feb 17, 2026
@savenoch ☝️