Export/Import does not import certification details and Chart changes
srinisubramanian opened this issue · comments
Exporting a dashboard and importing it into another instance/landscape of Superset (where the dashboard was previously imported and created) prompts to overwrite and on confirmation imports. The dashboard changes (title etc) are imported but the certification data entered for a dashboard is not imported. Also any chart changes are not imported.
How to reproduce the bug
- Create a dashboard and add a chart. Export and Import it to another instance where the dashboard does not exist. Import is successful but the Dashboard Certification and Certified By details are not imported.
- Now make some change in the Dashboard like changing title and also changes in the include chart in the original instance like changing title or making changes to the chart itself. Export the dashboard and import it to the second system. The dashboard changes are imported but the Chart changes are NOT imported.
- For testing I exported only the chart and imported the chart and it works. Looks like the overwrite of dashboard works but overwrite of chart is missing.
Expected results
Chart changes are also imported
Actual results
Chart changes are NOT imported
Screenshots
If applicable, add screenshots to help explain your problem.
Environment
(please complete the following information):
- browser type and version: Chrome latest
- superset version: 1.5.0
- python version: 3.8.12
- node.js version: N/A
- any feature flags active: DASHBOARD_RBAC, VERSIONED_EXPORT, ENABLE_TEMPLATE_PROCESSING and DASHBOARD_CROSS_FILTERS
Checklist
Make sure to follow these steps before submitting your issue - thank you!
- I have checked the superset logs for python stacktraces and included it here as text if there are any.
- I have reproduced the issue with at least the latest released version of superset.
- I have checked the issue tracker for the same issue and I haven't found one similar.
Additional context
Add any other context about the problem here.
Possible issue; I traced this to superset/charts/commands/importers/v1/utils.py. Although the API call is made with overwtite=True, the call always receives False. Setting it as True in the function correctly imports and overwrites the charts.
Tracing back further to the file: superset/dashboards/commands/importers/v1/utils.py
All calls to _import() receives the right flag from the API. However all the calls to import_database(), import_dataset() and import_chart() have overwrite hardcoded to False. Only the call to import_dashboard() passes in the flag as input in the API.
@betodealmeida @EBoisseauSierra Any assistance on this greatly appreciated
@srinisubramanian it's by design that chart changes are not imported — there could be other dashboards using that chart, and we don't want to break them.
In the current workflow the overwrite affects only the imported asset, not the related ones. The behavior is identical if you import an existing chart with a modified dataset, for example. I know someone from Nielsen (@ofekisr maybe?) was working on a PR where you can choose what you want to overwrite, but I'm not sure of its status.
The missing certification data, OTOH, is a bug, we should fix that.
@betodealmeida Thanks for the clarification. So to get the whole dashboard imported from one system to another (we need to do it programmatically) is there any workaround other than hacking the code? The obvious alternative seems to be to loop through and also import chart using the API?
@srinisubramanian there's another option: https://github.com/apache/superset/blob/master/superset/importexport/api.py#L93
If you POST the ZIP file to /api/v1/assets/import then everything will be overwritten.
Thanks @betodealmeida let me try that.
but I even can't get zip files imported where it's saying "An error occurred while importing chart: Import chart failed for an unknown reason Please re-export your file and try importing again
Database driver for importing maybe not installed. Visit the Superset documentation page for installation instructions:here."
So what do I should do to resolve this problem, thank you.
@betodealmeida I did try this today. I exported the dashboard using the dashboard export API (not the assets/export) and imported using the assets/import API. All I get is a cryptic error:
2022-06-03 09:52:23,500:INFO:werkzeug - - [03/Jun/2022 09:52:23] "POST /api/v1/assets/import HTTP/1.0" 308 -
Error importing assets
2022-06-03 09:52:24,237:WARNING:superset.views.base:Error importing assets
Can I import dashboards in this way or it needs to be exported via the assets/export API? Issue is I dont want to export all dashboards but only selected dashboard.
@betodealmeida Got the chance to finally upgrade to 2.0. The assets/import end point still fails with the following exception:
CommandException
Traceback (most recent call last):
File "/home/xxxxx/.pyenv/versions/3.8.12/envs/superset/lib/python3.8/site-packages/flask/app.py", line 1517, in full_dispatch_request
rv = self.dispatch_request()
File "/home/xxxxx/.pyenv/versions/3.8.12/envs/superset/lib/python3.8/site-packages/flask/app.py", line 1503, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
File "/home/xxxxx/.pyenv/versions/3.8.12/envs/superset/lib/python3.8/site-packages/flask_appbuilder/security/decorators.py", line 98, in wraps
return f(self, *args, **kwargs)
File "/home/xxxxx/.pyenv/versions/3.8.12/envs/superset/lib/python3.8/site-packages/superset/utils/log.py", line 245, in wrapper
value = f(*args, **kwargs)
File "/home/xxxxx/.pyenv/versions/3.8.12/envs/superset/lib/python3.8/site-packages/superset/views/base_api.py", line 98, in wraps
return f(self, *args, **kwargs)
File "/home/xxxxx/.pyenv/versions/3.8.12/envs/superset/lib/python3.8/site-packages/superset/importexport/api.py", line 161, in import_
command.run()
File "/home/xxxxx/.pyenv/versions/3.8.12/envs/superset/lib/python3.8/site-packages/superset/commands/importers/v1/assets.py", line 136, in run
self.validate()
File "/home/xxxxx/.pyenv/versions/3.8.12/envs/superset/lib/python3.8/site-packages/superset/commands/importers/v1/assets.py", line 164, in validate
raise exception
superset.commands.exceptions.CommandInvalidError: Error importing assets
This happens for both a new dashboard and an existing dashboard. However the dashboard/import endpoint works for a new dashboard. And as you had explained existing dashboards new charts are imported but changes to old ones are not overwritten.
Any thoughts on how to get the assets/import to work for a single dashboard
@betodealmeida One question. The file is being exported using dashboard/export since I wanted to export only 1 dashboard (and assets/export seems to export all with no filter), I extracted the zip file, modified metadata.yaml and changed type to assets (it was Dashboard). The modified zip file was imported successfully with assets/import.
Q1. is it ok to import the dashboard export as assets export as I see no change in the files after comparison
Q2. Is there a way to only assets/export selected dashboards?
I decided to use the export and import APIs of each entity ie. Dashboard, Database, Dataset and Chart. This works with overwrite option too.
@srinisubramanian can you please tell how you solved this issue
Please see previous comment. I started using individual import and export APIs to import each entity separately
is this issue was fixed? I'm strangling with the same nightmare here... @srinisubramanian @DharmenderKaushik
When changing dataset of a chart the import fails even with overwrite option
It was confirmed that overwrite works only for that entity. While importing dashboard it will only overwrite the dashboard. This is by design. If you want to overwrite dataset you need to import dataset with overwrite
oh, @srinisubramanian thanks for the immediate answer!
if I would like to copy the environment with export-assets every time to another environment (from dev to stagging or prod) and the change that I did is to change the chart's dataset (not a change in the dataset itself but assigning another dataset to exists chart)
I had the exact same requirement. See above comment from @betodealmeida
I tried it, unfortunately it didn't solve the problem.
Hello all @betodealmeida @SharonCastel @srinisubramanian , I opened SIP for purposing importing-item's ability to overwrite cascading items inside here #24660 I hope we could discuss further or sharing if there is any concerns for developing this ability.