Create more specifc error messages for invalid GTFS
grant-humphries opened this issue · comments
I recently added a new GTFS file to one of TriMet's OTP test instances (which uses OneBusAway's GTFS reader). The GTFS was invalid because I had misnamed a couple of the headings (within calendar.txt I had given the columns that should be labeled "start_date" and "end_date", the headings "date_on" and "date_off"), but the error message that was returned was not indicative of this.
Here's the output from when the invalid GTFS was read:
[INFO] 2012-11-13 17:24:40,556 INFO [DownloadableGtfsInputSource.java:96] : downloading gtfs: url=http://maps5.trimet.org/ferry_canby_ gtfs.zip path=/tmp/maps5.trimet.org_ferry_canby_gtfs.zip_gtfs.zip
[INFO] 2012-11-13 17:24:40,558 INFO [GtfsGraphBuilderImpl.java:205] : reading entities: org.onebusaway.gtfs.model.ShapePoint
[INFO] 2012-11-13 17:24:40,559 INFO [GtfsGraphBuilderImpl.java:205] : reading entities: org.onebusaway.gtfs.model.Route
[INFO] 2012-11-13 17:24:40,560 INFO [GtfsGraphBuilderImpl.java:205] : reading entities: org.onebusaway.gtfs.model.Stop
[INFO] 2012-11-13 17:24:40,561 INFO [GtfsGraphBuilderImpl.java:205] : reading entities: org.onebusaway.gtfs.model.Trip
[INFO] 2012-11-13 17:24:40,562 INFO [GtfsGraphBuilderImpl.java:205] : reading entities: org.onebusaway.gtfs.model.StopTime
[INFO] 2012-11-13 17:24:40,563 INFO [GtfsGraphBuilderImpl.java:205] : reading entities: org.onebusaway.gtfs.model.ServiceCalendar
[INFO] Exception in thread "main" org.onebusaway.csv_entities.exceptions.CsvEntityIOException: io error: entityType=org.onebusaway.gtfs .model.ServiceCalendar path=java.io.InputStreamReader@52c8c6d9 lineNumber=2
[INFO] at org.onebusaway.csv_entities.CsvEntityReader.readEntities(CsvEntityReader.java:161)
[INFO] at org.onebusaway.csv_entities.CsvEntityReader.readEntities(CsvEntityReader.java:120)
[INFO] at org.onebusaway.csv_entities.CsvEntityReader.readEntities(CsvEntityReader.java:115)
[INFO] at org.onebusaway.csv_entities.CsvEntityReader.readEntities(CsvEntityReader.java:108)
[INFO] at org.opentripplanner.graph_builder.impl.GtfsGraphBuilderImpl.loadBundle(GtfsGraphBuilderImpl.java:206)
[INFO] [INFO] 2012-11-13 17:24:40,566 INFO [AbstractApplicationContext.java:1002] : Closing org.springframework.context.support.Generi cApplicationContext@32c8f6f: startup date [Tue Nov 13 17:23:53 PST 2012]; root of context hierarchy
at org.opentripplanner.graph_builder.impl.GtfsGraphBuilderImpl.buildGraph(GtfsGraphBuilderImpl.java:142)
[INFO] at org.opentripplanner.graph_builder.GraphBuilderTask.run(GraphBuilderTask.java:133)
[INFO] at org.opentripplanner.graph_builder.GraphBuilderMain.main(GraphBuilderMain.java:49)
[INFO] at org.opentripplanner.integration.graph_builder.GraphBuilderIntegrationMain.main(GraphBuilderIntegrationMain.java:40)
[INFO] Caused by: org.onebusaway.csv_entities.exceptions.MissingRequiredFieldException: missing required field: start_date
[INFO] 2012-11-13 17:24:40,566 INFO [DefaultSingletonBeanRegistry.java:422] : Destroying singletons in org.springframework.beans.facto ry.support.DefaultListableBeanFactory@7e3bc47: defining beans [org.springframework.context.annotation.internalConfigurationAnnotationP rocessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalReq uiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor,graphService,graphBuilderTask,org.spr ingframework.beans.factory.config.CustomEditorConfigurer#0,gtfsBuilder,osmBuilder,nedBuilder,transitStreetLink,checkGeometry,mapBuilder ,raptorDataBuilder,transitLocalStreets]; root of factory hierarchy
[INFO] at org.onebusaway.csv_entities.schema.AbstractFieldMapping.isMissingAndOptional(AbstractFieldMapping.java:94)
[INFO] at org.onebusaway.gtfs.serialization.mappings.ServiceDateFieldMappingFactory$FieldMappingImpl.translateFromCSVToObject(ServiceD ateFieldMappingFactory.java:50)
[INFO] at org.onebusaway.csv_entities.IndividualCsvEntityReader.readEntity(IndividualCsvEntityReader.java:129)
[INFO] at org.onebusaway.csv_entities.IndividualCsvEntityReader.handleLine(IndividualCsvEntityReader.java:96)
[INFO] at org.onebusaway.csv_entities.CsvEntityReader.readEntities(CsvEntityReader.java:157)
[INFO] ... 8 more
[INFO] ------------------------------------------------------------------------
[ERROR] BUILD ERROR
[INFO] ------------------------------------------------------------------------
[INFO] Result of /bin/sh -c cd /home/bh/OpenTripPlanner/opentripplanner-integration && java -Xmx2900m -classpath /home/bh/OpenTripPlann er/opentripplanner-integration/target/classes:/home/bh/OpenTripPlanner/opentripplanner-graph-builder/target/opentripplanner-graph-build er-0.9.2-SNAPSHOT.jar:/home/bh/OpenTripPlanner/opentripplanner-routing/target/opentripplanner-routing-0.9.2-SNAPSHOT.jar:/home/bh/.m2/r epository/com/vividsolutions/jts/1.12/jts-1.12.jar:/home/bh/.m2/repository/xerces/xercesImpl/2.4.0/xercesImpl-2.4.0.jar:/home/bh/.m2/re pository/org/hibernate/hibernate-core/3.3.2.GA/hibernate-core-3.3.2.GA.jar:/home/bh/.m2/repository/antlr/antlr/2.7.6/antlr-2.7.6.jar:/h ome/bh/.m2/repository/commons-collections/commons-collections/3.1/commons-collections-3.1.jar:/home/bh/.m2/repository/dom4j/dom4j/1.6.1 /dom4j-1.6.1.jar:/home/bh/.m2/repository/xml-apis/xml-apis/1.0.b2/xml-apis-1.0.b2.jar:/home/bh/.m2/repository/javax/transaction/jta/1.1 /jta-1.1.jar:/home/bh/.m2/repository/org/slf4j/slf4j-api/1.5.8/slf4j-api-1.5.8.jar:/home/bh/.m2/repository/javassist/javassist/3.8.0.GA /javassist-3.8.0.GA.jar:/home/bh/.m2/repository/org/xerial/sqlite-jdbc/3.6.16/sqlite-jdbc-3.6.16.jar:/home/bh/.m2/repository/org/xerial /thirdparty/nestedvm/1.0/nestedvm-1.0.jar:/home/bh/.m2/repository/org/onebusaway/onebusaway-gtfs/1.3.0/onebusaway-gtfs-1.3.0.jar:/home/ bh/.m2/repository/org/onebusaway/onebusaway-csv-entities/1.1.0/onebusaway-csv-entities-1.1.0.jar:/home/bh/.m2/repository/commons-beanut ils/commons-beanutils/1.7.0/commons-beanutils-1.7.0.jar:/home/bh/.m2/repository/commons-logging/commons-logging/1.1.1/commons-logging-1 .1.1.jar:/home/bh/.m2/repository/org/springframework/spring-context/3.0.5.RELEASE/spring-context-3.0.5.RELEASE.jar:/home/bh/.m2/reposit ory/org/springframework/spring-aop/3.0.5.RELEASE/spring-aop-3.0.5.RELEASE.jar:/home/bh/.m2/repository/aopalliance/aopalliance/1.0/aopal liance-1.0.jar:/home/bh/.m2/repository/org/springframework/spring-asm/3.0.5.RELEASE/spring-asm-3.0.5.RELEASE.jar:/home/bh/.m2/repositor y/org/springframework/spring-beans/3.0.5.RELEASE/spring-beans-3.0.5.RELEASE.jar:/home/bh/.m2/repository/org/springframework/spring-core /3.0.5.RELEASE/spring-core-3.0.5.RELEASE.jar:/home/bh/.m2/repository/org/springframework/spring-expression/3.0.5.RELEASE/spring-express ion-3.0.5.RELEASE.jar:/home/bh/OpenTripPlanner/opentripplanner-utils/target/opentripplanner-utils-0.9.2-SNAPSHOT.jar:/home/bh/.m2/repos itory/org/geotools/gt-referencing/8.0-M4/gt-referencing-8.0-M4.jar:/home/bh/.m2/repository/java3d/vecmath/1.3.2/vecmath-1.3.2.jar:/home /bh/.m2/repository/commons-pool/commons-pool/1.5.4/commons-pool-1.5.4.jar:/home/bh/.m2/repository/org/geotools/gt-metadata/8.0-M4/gt-me tadata-8.0-M4.jar:/home/bh/.m2/repository/org/geotools/gt-opengis/8.0-M4/gt-opengis-8.0-M4.jar:/home/bh/.m2/repository/net/java/dev/jsr -275/jsr-275/1.0-beta-2/jsr-275-1.0-beta-2.jar:/home/bh/.m2/repository/javax/media/jai_core/1.1.3/jai_core-1.1.3.jar:/home/bh/.m2/repos itory/org/slf4j/slf4j-log4j12/1.5.8/slf4j-log4j12-1.5.8.jar:/home/bh/.m2/repository/log4j/log4j/1.2.14/log4j-1.2.14.jar:/home/bh/.m2/re pository/org/apache/httpcomponents/httpclient/4.1.2/httpclient-4.1.2.jar:/home/bh/.m2/repository/org/apache/httpcomponents/httpcore/4.1 .2/httpcore-4.1.2.jar:/home/bh/.m2/repository/commons-codec/commons-codec/1.3/commons-codec-1.3.jar:/home/bh/.m2/repository/org/apache/ commons/commons-math3/3.0/commons-math3-3.0.jar:/home/bh/.m2/repository/com/google/guava/guava/11.0.1/guava-11.0.1.jar:/home/bh/.m2/rep ository/com/google/code/findbugs/jsr305/1.3.9/jsr305-1.3.9.jar:/home/bh/.m2/repository/com/fasterxml/jackson/module/jackson-module-jaxb -annotations/2.0.5/jackson-module-jaxb-annotations-2.0.5.jar:/home/bh/.m2/repository/com/fasterxml/jackson/core/jackson-core/2.0.5/jack son-core-2.0.5.jar:/home/bh/.m2/repository/com/fasterxml/jackson/core/jackson-databind/2.0.5/jackson-databind-2.0.5.jar:/home/bh/.m2/re pository/com/fasterxml/jackson/core/jackson-annotations/2.0.5/jackson-annotations-2.0.5.jar:/home/bh/.m2/repository/org/geotools/gt-geo json/8.0-M4/gt-geojson-8.0-M4.jar:/home/bh/.m2/repository/org/geotools/gt-main/8.0-M4/gt-main-8.0-M4.jar:/home/bh/.m2/repository/org/ge otools/gt-api/8.0-M4/gt-api-8.0-M4.jar:/home/bh/.m2/repository/jdom/jdom/1.0/jdom-1.0.jar:/home/bh/.m2/repository/com/googlecode/json-s imple/json-simple/1.1/json-simple-1.1.jar:/home/bh/OpenTripPlanner/openstreetmap/target/openstreetmap-0.9.2-SNAPSHOT.jar:/home/bh/.m2/r epository/org/apache/axis/axis/1.4/axis-1.4.jar:/home/bh/.m2/repository/org/apache/axis/axis-jaxrpc/1.4/axis-jaxrpc-1.4.jar:/home/bh/.m 2/repository/commons-discovery/commons-discovery/0.4/commons-discovery-0.4.jar:/home/bh/.m2/repository/org/apache/commons/commons-compr ess/1.0/commons-compress-1.0.jar:/home/bh/.m2/repository/crosby/binary/osmpbf/1.2.1/osmpbf-1.2.1.jar:/home/bh/.m2/repository/com/google /protobuf/protobuf-java/2.4.1/protobuf-java-2.4.1.jar:/home/bh/.m2/repository/org/springframework/spring-jdbc/3.0.5.RELEASE/spring-jdbc -3.0.5.RELEASE.jar:/home/bh/.m2/repository/org/springframework/spring-tx/3.0.5.RELEASE/spring-tx-3.0.5.RELEASE.jar:/home/bh/.m2/reposit ory/org/springframework/spring-orm/3.0.5.RELEASE/spring-orm-3.0.5.RELEASE.jar:/home/bh/.m2/repository/postgresql/postgresql/8.3-603.jdb c4/postgresql-8.3-603.jdbc4.jar:/home/bh/.m2/repository/mysql/mysql-connector-java/5.0.5/mysql-connector-java-5.0.5.jar:/home/bh/.m2/re pository/org/geotools/gt-arcgrid/8.0-M4/gt-arcgrid-8.0-M4.jar:/home/bh/.m2/repository/org/geotools/gt-coverage/8.0-M4/gt-coverage-8.0-M 4.jar:/home/bh/.m2/repository/javax/media/jai_imageio/1.1/jai_imageio-1.1.jar:/home/bh/.m2/repository/it/geosolutions/imageio-ext/image io-ext-tiff/1.1.2/imageio-ext-tiff-1.1.2.jar:/home/bh/.m2/repository/it/geosolutions/imageio-ext/imageio-ext-utilities/1.1.2/imageio-ex t-utilities-1.1.2.jar:/home/bh/.m2/repository/javax/media/jai_codec/1.1.3/jai_codec-1.1.3.jar:/home/bh/.m2/repository/org/jaitools/jt-z onalstats/1.2.0/jt-zonalstats-1.2.0.jar:/home/bh/.m2/repository/org/jaitools/jt-utils/1.2.0/jt-utils-1.2.0.jar:/home/bh/.m2/repository/ it/geosolutions/imageio-ext/imageio-ext-arcgrid/1.1.2/imageio-ext-arcgrid-1.1.2.jar:/home/bh/.m2/repository/it/geosolutions/imageio-ext /imageio-ext-streams/1.1.2/imageio-ext-streams-1.1.2.jar:/home/bh/.m2/repository/org/geotools/gt-shapefile/8.0-M4/gt-shapefile-8.0-M4.j ar:/home/bh/.m2/repository/org/geotools/gt-data/8.0-M4/gt-data-8.0-M4.jar:/home/bh/.m2/repository/org/geotools/gt-geotiff/8.0-M4/gt-geo tiff-8.0-M4.jar:/home/bh/.m2/repository/org/geotools/gt-epsg-hsql/8.0-M4/gt-epsg-hsql-8.0-M4.jar:/home/bh/.m2/repository/hsqldb/hsqldb/ 1.8.0.7/hsqldb-1.8.0.7.jar:/home/bh/.m2/repository/net/java/dev/jets3t/jets3t/0.8.1/jets3t-0.8.1.jar:/home/bh/.m2/repository/commons-ht tpclient/commons-httpclient/3.1/commons-httpclient-3.1.jar:/home/bh/.m2/repository/com/jamesmurty/utils/java-xmlbuilder/0.4/java-xmlbui lder-0.4.jar:/home/bh/.m2/repository/com/beust/jcommander/1.27/jcommander-1.27.jar:/home/bh/.m2/repository/net/sourceforge/javacsv/java csv/2.0/javacsv-2.0.jar:/home/bh/.m2/repository/commons-cli/commons-cli/1.2/commons-cli-1.2.jar:/home/bh/.m2/repository/org/codehaus/ja ckson/jackson-mapper-asl/1.9.2/jackson-mapper-asl-1.9.2.jar:/home/bh/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.9.2/jackson -core-asl-1.9.2.jar org.opentripplanner.integration.graph_builder.GraphBuilderIntegrationMain execution is: '1'.
[INFO] ------------------------------------------------------------------------
[INFO] For more information, run Maven with the -e switch
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 2 minutes 48 seconds
[INFO] Finished at: Tue Nov 13 17:24:40 PST 2012
[INFO] Final Memory: 194M/1553M
[INFO] ------------------------------------------------------------------------
The library threw two exceptions:
org.onebusaway.csv_entities.exceptions.CsvEntityIOException: io error: entityType=org.onebusaway.gtfs .model.ServiceCalendar path=java.io.InputStreamReader@52c8c6d9 lineNumber=2
Caused by: org.onebusaway.csv_entities.exceptions.MissingRequiredFieldException: missing required field: start_date
I'll admit the path=... value isn't super useful, so I've updated the CsvEntityReader class to return path="calendar.txt", which should be more useful.
Probably more direct to your question, could the exceptions be displayed in a more human-friendly way? Most definitely, but I think that's a task for OTP. The GTFS reader library throws structured exception types that can be used to display a more human friendly message. But ultimately, it's up to each user of the library to decide how to handle the exceptions. We could add some helper functions for converting exceptions into loggable strings if you think that'd be useful.
Either way, I'm going to mark this as closed for now.