Revision processing fails for personal and organization accounts
sglavoie opened this issue · comments
When I use data-cli, I want to be able to data push with that tool successfully so I can publish my dataset to DataHub.
DataHub shows the message Processing this revision has failed. when pushing a dataset. See Analysis section below for more on this problem.
Acceptance
- Any user can publish to DataHub without facing revision processing issues.
Tasks
- TBC
Analysis
Organization accounts are set up locally like so before attempting to push:
export DATAHUB_JSON=~/.config/datahub/config.json
Example content for ~/.config/datahub/config.json:
{
"token": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx.xxxxxxxxaWQiOiJ0ZXN0IiwicGVybWlzc2lvbnMiOiIqIiwic2VydmljZSI6Ixxxxxxxxxx.xxxxxxxxxxxxxxxx-xxxxxxxxxxxxxxxxxxxxxxxxxx",
"profile": {
"id": "test",
"username": "test"
}
}Personal accounts are set up similarly with export DATAHUB_JSON, the content of the configuration file being populated when running data login and logging in with Google.
data push succeeds and a new revision appears on DataHub (but fails to validate). Here's the output of data push --debug for a dataset that was successfully pushed before (e.g. see the dataset here https://datahub.io/core/unicode-emojis):
�[2K> [debug] Getting rawstore upload creds
> [debug] Getting authz token for rawstore service
> [debug] Calling rawstore authorize with
> [debug] {
"metadata": {
"owner": "test",
"findability": "published"
},
"filedata": {
"archive/emojis.csv": {
"length": 471617,
"md5": "PYg4sBctHL9wtlQhBA5TCQ==",
"name": "emojis_original"
},
"data/validation_report.json": {
"length": 626,
"md5": "BSKlIQLI08KZpw3gxusDAw==",
"name": "validation_report"
},
"data/emojis_csv.csv": {
"length": 471617,
"md5": "PYg4sBctHL9wtlQhBA5TCQ==",
"name": "emojis_csv"
},
"data/emojis_json.json": {
"length": 1026404,
"md5": "VJaQT6vB3k20jahoI5Id4Q==",
"name": "emojis_json"
},
"README.md": {
"length": 1576,
"md5": "34J2Qrj4SPPlnWdJzf+E5Q==",
"name": "readme"
},
"datapackage.json": {
"length": 6349,
"md5": "0gJzaT/cetPT8GHz64BF/w==",
"name": "datapackage.json"
}
}
}
> [debug] Uploading to rawstore with creds ...
> [debug] {
"archive/emojis.csv": {
"upload_url": "https://s3.amazonaws.com/rawstore.datahub.io",
"upload_query": {
"acl": "public-read",
"Content-MD5": "PYg4sBctHL9wtlQhBA5TCQ==",
"Content-Type": "text/plain",
"key": "3d8838b0172d1cbf70b65421040e5309.csv",
"x-amz-algorithm": "AWS4-HMAC-SHA256",
"x-amz-credential": "AKIAUCXJYH2UCIYCVEFM/20210202/us-east-1/s3/aws4_request",
"x-amz-date": "20210202T193529Z",
"policy": "eyJleHBpcmF0aW9uIjogIjIwMjEtMDItMDJUMjA6MzU6MjlaIiwgImNvbmRpdGlvbnMiOiBbeyJhY2wiOiAicHVibGljLXJlYWQifSwgeyJDb250ZW50LVR5cGUiOiAidGV4dC9wbGFpbiJ9LCB7IkNvbnRlbnQtTUQ1IjogIlBZZzRzQmN0SEw5d3RsUWhCQTVUQ1E9PSJ9LCB7ImJ1Y2tldCI6ICJyYXdzdG9yZS5kYXRhaHViLmlvIn0sIHsia2V5IjogIjNkODgzOGIwMTcyZDFjYmY3MGI2NTQyMTA0MGU1MzA5LmNzdiJ9LCB7IngtYW16LWFsZ29yaXRobSI6ICJBV1M0LUhNQUMtU0hBMjU2In0sIHsieC1hbXotY3JlZGVudGlhbCI6ICJBS0lBVUNYSllIMlVDSVlDVkVGTS8yMDIxMDIwMi91cy1lYXN0LTEvczMvYXdzNF9yZXF1ZXN0In0sIHsieC1hbXotZGF0ZSI6ICIyMDIxMDIwMlQxOTM1MjlaIn1dfQ==",
"x-amz-signature": "77689e63b628d81aac4fc5e65061adb91a8c21ac90c776d6933a0ea78e221993"
},
"exists": true
},
"data/validation_report.json": {
"upload_url": "https://s3.amazonaws.com/rawstore.datahub.io",
"upload_query": {
"acl": "public-read",
"Content-MD5": "BSKlIQLI08KZpw3gxusDAw==",
"Content-Type": "text/plain",
"key": "0522a52102c8d3c299a70de0c6eb0303.json",
"x-amz-algorithm": "AWS4-HMAC-SHA256",
"x-amz-credential": "AKIAUCXJYH2UCIYCVEFM/20210202/us-east-1/s3/aws4_request",
"x-amz-date": "20210202T193529Z",
"policy": "eyJleHBpcmF0aW9uIjogIjIwMjEtMDItMDJUMjA6MzU6MjlaIiwgImNvbmRpdGlvbnMiOiBbeyJhY2wiOiAicHVibGljLXJlYWQifSwgeyJDb250ZW50LVR5cGUiOiAidGV4dC9wbGFpbiJ9LCB7IkNvbnRlbnQtTUQ1IjogIkJTS2xJUUxJMDhLWnB3M2d4dXNEQXc9PSJ9LCB7ImJ1Y2tldCI6ICJyYXdzdG9yZS5kYXRhaHViLmlvIn0sIHsia2V5IjogIjA1MjJhNTIxMDJjOGQzYzI5OWE3MGRlMGM2ZWIwMzAzLmpzb24ifSwgeyJ4LWFtei1hbGdvcml0aG0iOiAiQVdTNC1ITUFDLVNIQTI1NiJ9LCB7IngtYW16LWNyZWRlbnRpYWwiOiAiQUtJQVVDWEpZSDJVQ0lZQ1ZFRk0vMjAyMTAyMDIvdXMtZWFzdC0xL3MzL2F3czRfcmVxdWVzdCJ9LCB7IngtYW16LWRhdGUiOiAiMjAyMTAyMDJUMTkzNTI5WiJ9XX0=",
"x-amz-signature": "7a459a6d81ddb9536852715d2d749e9c78bd3e93406907b536e7bc8ba90d332b"
},
"exists": true
},
"data/emojis_csv.csv": {
"upload_url": "https://s3.amazonaws.com/rawstore.datahub.io",
"upload_query": {
"acl": "public-read",
"Content-MD5": "PYg4sBctHL9wtlQhBA5TCQ==",
"Content-Type": "text/plain",
"key": "3d8838b0172d1cbf70b65421040e5309.csv",
"x-amz-algorithm": "AWS4-HMAC-SHA256",
"x-amz-credential": "AKIAUCXJYH2UCIYCVEFM/20210202/us-east-1/s3/aws4_request",
"x-amz-date": "20210202T193529Z",
"policy": "eyJleHBpcmF0aW9uIjogIjIwMjEtMDItMDJUMjA6MzU6MjlaIiwgImNvbmRpdGlvbnMiOiBbeyJhY2wiOiAicHVibGljLXJlYWQifSwgeyJDb250ZW50LVR5cGUiOiAidGV4dC9wbGFpbiJ9LCB7IkNvbnRlbnQtTUQ1IjogIlBZZzRzQmN0SEw5d3RsUWhCQTVUQ1E9PSJ9LCB7ImJ1Y2tldCI6ICJyYXdzdG9yZS5kYXRhaHViLmlvIn0sIHsia2V5IjogIjNkODgzOGIwMTcyZDFjYmY3MGI2NTQyMTA0MGU1MzA5LmNzdiJ9LCB7IngtYW16LWFsZ29yaXRobSI6ICJBV1M0LUhNQUMtU0hBMjU2In0sIHsieC1hbXotY3JlZGVudGlhbCI6ICJBS0lBVUNYSllIMlVDSVlDVkVGTS8yMDIxMDIwMi91cy1lYXN0LTEvczMvYXdzNF9yZXF1ZXN0In0sIHsieC1hbXotZGF0ZSI6ICIyMDIxMDIwMlQxOTM1MjlaIn1dfQ==",
"x-amz-signature": "77689e63b628d81aac4fc5e65061adb91a8c21ac90c776d6933a0ea78e221993"
},
"exists": true
},
"data/emojis_json.json": {
"upload_url": "https://s3.amazonaws.com/rawstore.datahub.io",
"upload_query": {
"acl": "public-read",
"Content-MD5": "VJaQT6vB3k20jahoI5Id4Q==",
"Content-Type": "text/plain",
"key": "5496904fabc1de4db48da86823921de1.json",
"x-amz-algorithm": "AWS4-HMAC-SHA256",
"x-amz-credential": "AKIAUCXJYH2UCIYCVEFM/20210202/us-east-1/s3/aws4_request",
"x-amz-date": "20210202T193529Z",
"policy": "eyJleHBpcmF0aW9uIjogIjIwMjEtMDItMDJUMjA6MzU6MjlaIiwgImNvbmRpdGlvbnMiOiBbeyJhY2wiOiAicHVibGljLXJlYWQifSwgeyJDb250ZW50LVR5cGUiOiAidGV4dC9wbGFpbiJ9LCB7IkNvbnRlbnQtTUQ1IjogIlZKYVFUNnZCM2syMGphaG9JNUlkNFE9PSJ9LCB7ImJ1Y2tldCI6ICJyYXdzdG9yZS5kYXRhaHViLmlvIn0sIHsia2V5IjogIjU0OTY5MDRmYWJjMWRlNGRiNDhkYTg2ODIzOTIxZGUxLmpzb24ifSwgeyJ4LWFtei1hbGdvcml0aG0iOiAiQVdTNC1ITUFDLVNIQTI1NiJ9LCB7IngtYW16LWNyZWRlbnRpYWwiOiAiQUtJQVVDWEpZSDJVQ0lZQ1ZFRk0vMjAyMTAyMDIvdXMtZWFzdC0xL3MzL2F3czRfcmVxdWVzdCJ9LCB7IngtYW16LWRhdGUiOiAiMjAyMTAyMDJUMTkzNTI5WiJ9XX0=",
"x-amz-signature": "7580cd6714890c0cc6c1f69ecfd0964d9da7a4fe6733e31abb191bf241b83161"
},
"exists": true
},
"README.md": {
"upload_url": "https://s3.amazonaws.com/rawstore.datahub.io",
"upload_query": {
"acl": "public-read",
"Content-MD5": "34J2Qrj4SPPlnWdJzf+E5Q==",
"Content-Type": "text/plain",
"key": "df827642b8f848f3e59d6749cdff84e5.md",
"x-amz-algorithm": "AWS4-HMAC-SHA256",
"x-amz-credential": "AKIAUCXJYH2UCIYCVEFM/20210202/us-east-1/s3/aws4_request",
"x-amz-date": "20210202T193529Z",
"policy": "eyJleHBpcmF0aW9uIjogIjIwMjEtMDItMDJUMjA6MzU6MjlaIiwgImNvbmRpdGlvbnMiOiBbeyJhY2wiOiAicHVibGljLXJlYWQifSwgeyJDb250ZW50LVR5cGUiOiAidGV4dC9wbGFpbiJ9LCB7IkNvbnRlbnQtTUQ1IjogIjM0SjJRcmo0U1BQbG5XZEp6ZitFNVE9PSJ9LCB7ImJ1Y2tldCI6ICJyYXdzdG9yZS5kYXRhaHViLmlvIn0sIHsia2V5IjogImRmODI3NjQyYjhmODQ4ZjNlNTlkNjc0OWNkZmY4NGU1Lm1kIn0sIHsieC1hbXotYWxnb3JpdGhtIjogIkFXUzQtSE1BQy1TSEEyNTYifSwgeyJ4LWFtei1jcmVkZW50aWFsIjogIkFLSUFVQ1hKWUgyVUNJWUNWRUZNLzIwMjEwMjAyL3VzLWVhc3QtMS9zMy9hd3M0X3JlcXVlc3QifSwgeyJ4LWFtei1kYXRlIjogIjIwMjEwMjAyVDE5MzUyOVoifV19",
"x-amz-signature": "d30a5efa4af4c0878a382b189a9c640529544c5731051406e7a54bd40136cff7"
},
"exists": true
},
"datapackage.json": {
"upload_url": "https://s3.amazonaws.com/rawstore.datahub.io",
"upload_query": {
"acl": "public-read",
"Content-MD5": "0gJzaT/cetPT8GHz64BF/w==",
"Content-Type": "text/plain",
"key": "d20273693fdc7ad3d3f061f3eb8045ff.json",
"x-amz-algorithm": "AWS4-HMAC-SHA256",
"x-amz-credential": "AKIAUCXJYH2UCIYCVEFM/20210202/us-east-1/s3/aws4_request",
"x-amz-date": "20210202T193529Z",
"policy": "eyJleHBpcmF0aW9uIjogIjIwMjEtMDItMDJUMjA6MzU6MjlaIiwgImNvbmRpdGlvbnMiOiBbeyJhY2wiOiAicHVibGljLXJlYWQifSwgeyJDb250ZW50LVR5cGUiOiAidGV4dC9wbGFpbiJ9LCB7IkNvbnRlbnQtTUQ1IjogIjBnSnphVC9jZXRQVDhHSHo2NEJGL3c9PSJ9LCB7ImJ1Y2tldCI6ICJyYXdzdG9yZS5kYXRhaHViLmlvIn0sIHsia2V5IjogImQyMDI3MzY5M2ZkYzdhZDNkM2YwNjFmM2ViODA0NWZmLmpzb24ifSwgeyJ4LWFtei1hbGdvcml0aG0iOiAiQVdTNC1ITUFDLVNIQTI1NiJ9LCB7IngtYW16LWNyZWRlbnRpYWwiOiAiQUtJQVVDWEpZSDJVQ0lZQ1ZFRk0vMjAyMTAyMDIvdXMtZWFzdC0xL3MzL2F3czRfcmVxdWVzdCJ9LCB7IngtYW16LWRhdGUiOiAiMjAyMTAyMDJUMTkzNTI5WiJ9XX0=",
"x-amz-signature": "d156ea79d155836ee5883a213ca4351fc4fc062248a5e60eb848c9f39970aed2"
},
"exists": true
}
}
> [debug] Uploads to rawstore: Complete
> [debug] Uploading to source spec store
> [debug] Getting authz token for rawstore service
> [debug] Calling source upload with spec
> [debug] {
"meta": {
"version": 1,
"ownerid": "test",
"owner": "test",
"dataset": "unicode-emojis",
"findability": "published"
},
"inputs": [
{
"kind": "datapackage",
"url": "https://s3.amazonaws.com/rawstore.datahub.io/d20273693fdc7ad3d3f061f3eb8045ff.json",
"parameters": {
"resource-mapping": {
"archive/emojis.csv": "https://s3.amazonaws.com/rawstore.datahub.io/3d8838b0172d1cbf70b65421040e5309.csv",
"data/emojis_csv.csv": "https://s3.amazonaws.com/rawstore.datahub.io/3d8838b0172d1cbf70b65421040e5309.csv",
"data/emojis_json.json": "https://s3.amazonaws.com/rawstore.datahub.io/5496904fabc1de4db48da86823921de1.json",
"data/validation_report.json": "https://s3.amazonaws.com/rawstore.datahub.io/0522a52102c8d3c299a70de0c6eb0303.json",
"README.md": "https://s3.amazonaws.com/rawstore.datahub.io/df827642b8f848f3e59d6749cdff84e5.md"
},
"descriptor": {
"bytes": 2443457,
"count_of_rows": 4590,
"datahub": {
"created": "2020-09-24T12:39:55.385717",
"findability": "published",
"flowid": "722d7c8098a51a04a1d3a5d41fc9f11c/unicode-emojis/2",
"modified": "2021-02-02T16:18:05.505508",
"owner": "sebastien.lavoie",
"ownerid": "722d7c8098a51a04a1d3a5d41fc9f11c"
},
"descriptor": "List of emojis available from the Unicode Consortium. More information can be found in the Unicode® Technical Standard #51.",
"hash": "60e1206a07ee5bb5e89dcc6b627f9fa5",
"id": "722d7c8098a51a04a1d3a5d41fc9f11c/unicode-emojis",
"keywords": [
"unicode",
"emojis",
"emoji",
"51",
"standard",
"uts"
],
"licenses": [
{
"name": "ODC-PDDL-1.0",
"path": "http://opendatacommons.org/licenses/pddl/",
"title": "Open Data Commons Public Domain Dedication and License v1.0"
}
],
"name": "unicode-emojis",
"profile": "data-package",
"resources": [
{
"path": "archive/emojis.csv",
"pathType": "local",
"name": "emojis_original",
"format": "csv",
"mediatype": "text/csv",
"bytes": 471617,
"datahub": {
"type": "original"
},
"dialect": {
"caseSensitiveHeader": false,
"delimiter": ",",
"doubleQuote": true,
"header": true,
"lineTerminator": "\r\n",
"quoteChar": "\"",
"skipInitialSpace": false
},
"dpp:streamedFrom": "https://s3.amazonaws.com/rawstore.datahub.io/3d8838b0172d1cbf70b65421040e5309.csv",
"encoding": "UTF-8",
"hash": "3d8838b0172d1cbf70b65421040e5309",
"profile": "tabular-data-resource",
"schema": {
"fields": [
{
"format": "default",
"name": "Group",
"type": "string"
},
{
"format": "default",
"name": "Subgroup",
"type": "string"
},
{
"format": "default",
"name": "CodePoint",
"type": "string"
},
{
"format": "default",
"name": "Status",
"type": "string"
},
{
"format": "default",
"name": "Representation",
"type": "string"
},
{
"format": "default",
"name": "Name",
"type": "string"
},
{
"format": "default",
"name": "Section",
"type": "string"
}
],
"missingValues": [
""
]
}
},
{
"path": "data/validation_report.json",
"pathType": "local",
"name": "validation_report",
"format": "json",
"mediatype": "application/json",
"bytes": 626,
"datahub": {
"type": "derived/report"
},
"description": "Validation report for tabular data",
"dpp:streamedFrom": "/tmp/tmpjfdpj8_x/722d7c8098a51a04a1d3a5d41fc9f11c/unicode-emojis/validation_report/data/0522a52102c8d3c299a70de0c6eb0303/validation_report.json",
"hash": "0522a52102c8d3c299a70de0c6eb0303",
"profile": "data-resource",
"encoding": "ISO-8859-1"
},
{
"path": "data/emojis_csv.csv",
"pathType": "local",
"name": "emojis_csv",
"format": "csv",
"mediatype": "text/csv",
"bytes": 471617,
"datahub": {
"derivedFrom": [
"emojis"
],
"type": "derived/csv"
},
"delimiter": ",",
"dialect": {
"caseSensitiveHeader": false,
"delimiter": ",",
"doubleQuote": true,
"header": true,
"lineTerminator": "\r\n",
"quoteChar": "\"",
"skipInitialSpace": false
},
"dpp:streamedFrom": "/tmp/tmpjfdpj8_x/722d7c8098a51a04a1d3a5d41fc9f11c/unicode-emojis/emojis_csv/data/2ed376f1c761e6d2936435e1addbf0ca/emojis_csv.csv",
"encoding": "UTF-8",
"hash": "2ed376f1c761e6d2936435e1addbf0ca",
"profile": "tabular-data-resource",
"quotechar": "\"",
"rowcount": 4590,
"schema": {
"fields": [
{
"format": "default",
"name": "Group",
"type": "string"
},
{
"format": "default",
"name": "Subgroup",
"type": "string"
},
{
"format": "default",
"name": "CodePoint",
"type": "string"
},
{
"format": "default",
"name": "Status",
"type": "string"
},
{
"format": "default",
"name": "Representation",
"type": "string"
},
{
"format": "default",
"name": "Name",
"type": "string"
},
{
"format": "default",
"name": "Section",
"type": "string"
}
],
"missingValues": [
""
]
},
"signed": true
},
{
"path": "data/emojis_json.json",
"pathType": "local",
"name": "emojis_json",
"format": "json",
"mediatype": "text/csv",
"bytes": 1026404,
"datahub": {
"derivedFrom": [
"emojis"
],
"type": "derived/json"
},
"delimiter": ",",
"dialect": {
"caseSensitiveHeader": false,
"delimiter": ",",
"doubleQuote": true,
"header": true,
"lineTerminator": "\r\n",
"quoteChar": "\"",
"skipInitialSpace": false
},
"dpp:streamedFrom": "/tmp/tmpjfdpj8_x/722d7c8098a51a04a1d3a5d41fc9f11c/unicode-emojis/emojis_json/data/5496904fabc1de4db48da86823921de1/emojis_json.json",
"encoding": "ISO-8859-1",
"hash": "5496904fabc1de4db48da86823921de1",
"profile": "tabular-data-resource",
"quotechar": "\"",
"rowcount": 4590,
"schema": {
"fields": [
{
"format": "default",
"name": "Group",
"type": "string"
},
{
"format": "default",
"name": "Subgroup",
"type": "string"
},
{
"format": "default",
"name": "CodePoint",
"type": "string"
},
{
"format": "default",
"name": "Status",
"type": "string"
},
{
"format": "default",
"name": "Representation",
"type": "string"
},
{
"format": "default",
"name": "Name",
"type": "string"
},
{
"format": "default",
"name": "Section",
"type": "string"
}
],
"missingValues": [
""
]
},
"signed": true
},
{
"path": "README.md",
"pathType": "local",
"name": "readme",
"format": "md",
"mediatype": "text/markdown",
"bytes": 1576,
"dpp:streamedFrom": "/tmp/tmpotcydkuk/README.md",
"encoding": "UTF-8"
}
],
"sources": [
{
"name": "unicode-emoji",
"path": "https://unicode.org/Public/emoji/latest/emoji-test.txt",
"title": "UTS #51 Unicode Emoji"
}
],
"title": "UTS #51 Unicode Emoji",
"readme": "List of emojis available from the Unicode Consortium. More information can be found in the [Unicode® Technical Standard #51](https://unicode.org/reports/tr51/).\n\n# Data\n\nThe data comes from [this directory at Unicode.org](https://unicode.org/Public/emoji/latest/) as a set of plain text files.\n\n\n# Sources\n\n- The Unicode Consortium: https://unicode.org/Public/emoji/latest/\n\n# Preparation\n\nPython 3.6+ is required. First, install dependencies with the following command:\n\n pip install -r scripts/requirements.txt\n\n\nThen, run the script from this directory in the following way:\n\n python scripts/process.py\n\n\n# License\n\nThis Data Package is licensed by its maintainers under the Public Domain Dedication and License (PDDL).\n\n> Copyright © 1991-2017 Unicode, Inc. All rights reserved.\n> Distributed under the Terms of Use in http://www.unicode.org/copyright.html.\n> \n> Permission is hereby granted, free of charge, to any person obtaining a copy of the Unicode data files and any associated documentation (the \"Data Files\") or Unicode software and any associated documentation (the \"Software\") to deal in the Data Files or Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, and/or sell copies of the Data Files or Software, and to permit persons to whom the Data Files or Software are furnished to do so, provided that either (a) this copyright and permission notice appear with all copies of the Data Files or Software, or (b) this copyright and permission notice appear in associated Documentation.\n"
}
}
}
],
"processing": [
{
"input": "emojis_original",
"output": "emojis_original",
"tabulator": {
"delimiter": ",",
"quotechar": "\""
}
},
{
"input": "emojis_csv",
"output": "emojis_csv",
"tabulator": {
"delimiter": ",",
"quotechar": "\""
}
},
{
"input": "emojis_json",
"output": "emojis_json",
"tabulator": {
"delimiter": ",",
"quotechar": "\""
}
}
]
}
> [debug] Getting authz token for source service
> [debug] {
"dataset_id": "test/unicode-emojis",
"errors": [],
"flow_id": "test/unicode-emojis/4",
"success": true
}
🙌 your data is published!
🔗 https://datahub.io/test/unicode-emojis/v/4 (copied to clipboard)
Opening the resources, e.g. https://s3.amazonaws.com/rawstore.datahub.io/d20273693fdc7ad3d3f061f3eb8045ff.json shows they are successfully uploaded. However, DataHub shows this:
https://datahub.io/sebastien.lavoie/unicode-emojis/v/7 (the code from this error can be found here https://github.com/datopian/assembler/blob/master/datapackage_pipelines_assembler/processors/load_private_resource.py#L19)
Processing this revision has failed. See below for details:
Copying source data
assembler.load_private_resource
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/datapackage_pipelines_assembler/processors/load_private_resource.py", line 35, in <module>
PrivateResourceLoader()()
File "/usr/lib/python3.6/site-packages/datapackage_pipelines_assembler/processors/load_private_resource.py", line 19, in __init__
if PKGSTORE_BUCKET in url:
TypeError: argument of type 'NoneType' is not iterable
This is not a one-time occurrence as can be seen here:
- https://datahub.io/sebastien.lavoie/unicode-emojis/v/3
- https://datahub.io/sebastien.lavoie/unicode-emojis/v/4
- https://datahub.io/sebastien.lavoie/unicode-emojis/v/6
- https://datahub.io/test/unicode-emojis/v/1
- https://datahub.io/test/unicode-emojis/v/2
- https://datahub.io/test/unicode-emojis/v/3
- https://datahub.io/test/unicode-emojis/v/4
- https://datahub.io/core/unicode-emojis/v/193
- https://datahub.io/core/unicode-emojis/v/194
For some unknown reason, https://datahub.io/sebastien.lavoie/unicode-emojis/v/5 worked while /v/3, /v/4 and /v/6 didn't.
Also, https://datahub.io/core/unicode-emojis/v/194 presented an additional error (appearing twice in the output on DataHub), for which the code can be found here https://github.com/datopian/assembler/blob/master/datapackage_pipelines_assembler/processors/dump_to_s3.py#L27 :
assembler.dump_to_s3
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/datapackage_pipelines_assembler/processors/dump_to_s3.py", line 32, in <module>
MyS3Dumper()()
File "/usr/lib/python3.6/site-packages/datapackage_pipelines/lib/dump/dumper_base.py", line 47, in __call__
finalizer=self.finalize
File "/usr/lib/python3.6/site-packages/datapackage_pipelines/wrapper/wrapper.py", line 64, in spew
for res in resources_iterator:
File "/usr/lib/python3.6/site-packages/datapackage_pipelines/lib/dump/dumper_base.py", line 150, in handle_resources
self.handle_datapackage(datapackage, parameters, stats)
File "/usr/lib/python3.6/site-packages/datapackage_pipelines/lib/dump/dumper_base.py", line 212, in handle_datapackage
location = self.write_file_to_output(temp_file_name, 'datapackage.json')
File "/usr/lib/python3.6/site-packages/datapackage_pipelines_aws/s3_dumper.py", line 46, in write_file_to_output
Key=key)
File "/usr/lib/python3.6/site-packages/datapackage_pipelines_assembler/processors/dump_to_s3.py", line 27, in put_object
datetime.datetime.now()
File "/usr/lib/python3.6/site-packages/filemanager/models.py", line 103, in add_file
assert sf.owner == owner
AssertionError
After having failed for about 2 hours straight, data push was successful here for a personal account https://datahub.io/sebastien.lavoie/unicode-emojis/v/9 as well as here https://datahub.io/core/unicode-emojis for an organization account.
At this moment, the issue seems to be no longer reproducible.
INVALID. This relates to datahub-v2 which is now deprecated.