Custom Import Form: Bug when form fields are also in Resource.import_id_field

[Dave White]

Description
When using a custom import form AND any of those custom form fields are part of the import_id_fields (because those fields are a compound key) in the corresponding custom resource in admin.py, all rows in the csv will be falsely classified as new instances instead of existing instances, thus resulting in a database error due to key violation.

Theory:

• django-import-export/import_export/resources.py line 777:
  • The argument 'row' being passed to self.get_or_init_instance(instance_loader, row) is not including the form fields/values in the custom import form, thus not representing truly what is to be inserted since the custom form field data will be inserted also for each row.
  • If any of those fields from the custom import form happen to also be in the import_id_fields then when Resource.get_instance() is called, it will not detect an existing instance (resources.py line 390) and will return 'None', and so in turn, get_or_init_instance() is treating that row as a new instance instead of existing.
  • So the fix would be to make sure that the form fields are added to the 'row', so that get_instance returns correctly

To Reproduce
Steps to reproduce the behavior:

1. Implement a custom import form as per documentation
2. Make sure at least one of the fields in the custom import form is part of each row's primary key and thus would also exist in the resources.ModelResource.Meta.import_id_fields
3. Submit a .csv with records that already exist in the application
4. Receive a response that is passed back from the database regarding a duplicate key error.

Versions (please complete the following information):

• Django Import Export: 3.3.6
• Python 3.9.1
• Django 4.2.9

Expected behavior
User should be presented with an expected update view which would show which fields will change on the existing records.

Additional context
I discovered this because I have implemented a custom import form that contains two fields, both of which are also in the import_id_fields list in the custom resource (there are 4 fields total in import_id_fields, the other two fields are in the csv).

[Matt Hegarty]

Thanks for submitting. It sounds likely that the get_or_init_instance is not detecting your form values, because it would have no way of knowing that these are part of import_id_fields. Since you are extending logic using forms, I would suggest this would require additional logic on your part, rather than this being in a bug in the core library, although I haven't looked into this in detail yet. I think the fix would be to add the fields to the row as you suggest.

I would suggest switching to v4 beta if you can because all future dev will be in that release from this year.

It would be ideal if you could

• Test you still get the issue when using v4 beta (it's likely that you will but is perhaps worth testing)
• A reproducible example using the example app would be ideal.

[Dave White]

Thanks @matthewhegarty. I will look into setting up an environment so I may test the v4 beta and see if I can make a test example of this use case.

Re: is django-import-export is seeing the form fields in my own implementation, I believe yes because 1) When I submit a CSV with new records (i.e. new instances), the form field values are viewable in the Confirm Import screen appended to the CSV values as part of the overall record, and everything inserts correctly. Also, 2) I see the form values in the debugger as I step through.