Credential/training screwy handling of SYSTEM_MAINTENANCE_NO_UPLOAD

[bemoody]

credential_application raises ServiceUnavailable if SYSTEM_MAINTENANCE_NO_UPLOAD is set. I think this is wrong because I think this form does not upload files anymore.

On the other hand, edit_trainings does not pay attention to SYSTEM_MAINTENANCE_NO_UPLOAD. This is broken and dangerous.

See also issue #2051

[bemoody]

Anyway, it would probably be useful to have an option to allow new uploads of training applications without allowing changes to project files.

[bemoody]

Also note the PAUSE_CREDENTIALING variable, which is somewhat related though intended for a different purpose.

PAUSE_CREDENTIALING causes the link to the credentialing form to be hidden - it doesn't prevent submitting the form. (This is a good thing IMHO; it may have been intentional.)

Perhaps PAUSE_CREDENTIALING should also cause the training submission form to be hidden.

Another issue is that "no changes" mode should probably imply "no uploads". Maybe "no changes" should also imply "pause credentialing" (or maybe not.)

[bemoody]

Okay, so I had the thought that it might be better to add the behavior to django.db.models.FileField instead of to the views.

So basically I did this

diff --git a/physionet-django/physionet/settings/base.py b/physionet-django/physionet/settings/base.py
index 4035feb03..437d4ebab 100644
--- a/physionet-django/physionet/settings/base.py
+++ b/physionet-django/physionet/settings/base.py
@@ -606,7 +606,8 @@ class StorageTypes:
     STATICFILES_STORAGE = 'physionet.storage.StaticStorage'
     GCP_BUCKET_LOCATION = config('GCP_BUCKET_LOCATION')
     GS_PROJECT_ID = config('GCP_PROJECT_ID')
-
+else:
+    DEFAULT_FILE_STORAGE = 'physionet.storage.LocalMediaStorage'
 
 # Cloud research environment integration
 # See: https://pypi.org/project/hdn-research-environment/
diff --git a/physionet-django/physionet/storage.py b/physionet-django/physionet/storage.py
index 965ce44a1..751c3ab32 100644
--- a/physionet-django/physionet/storage.py
+++ b/physionet-django/physionet/storage.py
@@ -1,9 +1,33 @@
 import datetime as dt
 
 from django.conf import settings
+from django.core.files.storage import FileSystemStorage
 from storages.backends.gcloud import GoogleCloudStorage
 
+from physionet.middleware.maintenance import ServiceUnavailable
 
+
+class StorageMaintenanceMixin:
+    def _open(self, name, mode='rb'):
+        if settings.SYSTEM_MAINTENANCE_NO_UPLOAD and not mode.startswith('r'):  # oops, this is wrong, but anyway
+            raise ServiceUnavailable()
+        return super()._open(name, mode)
+
+    def _save(self, name, content):
+        if settings.SYSTEM_MAINTENANCE_NO_UPLOAD:
+            raise ServiceUnavailable()
+        return super()._save(name, content)
+
+    def delete(self, name):
+        if settings.SYSTEM_MAINTENANCE_NO_UPLOAD:
+            raise ServiceUnavailable()
+        return super().delete(name)
+
+
+class LocalMediaStorage(StorageMaintenanceMixin, FileSystemStorage):
+    pass
+
+
 class MediaStorage(GoogleCloudStorage):
     bucket_name = settings.GCP_STORAGE_BUCKET_NAME
     location = ''

and that seems to work, maybe? But then:

• The maint middleware disables itself by default, so it doesn't work for temporarily setting NO_UPLOAD in the test suite.
• If I get rid of the MiddlewareNotUsed, then the middleware runs, but service_unavailable() raises a TransactionManagementError.
• Then, it seems like django tries to render a 500 error page, and that raises a TransactionManagementError.

Not clear if this is some screwiness in Django testing that I need to work around, or if it's some inherent screwiness about database transactions that I don't understand. (Shouldn't the failed transaction be dead and gone at this point?)

But surely, rendering the 500 page ought to work even if the database goes completely pear-shaped, right?! Why is the template renderer poking the database at all?

On top of all that, it's not obvious to me whether django.db.models.FileField actually does anything to handle exceptions while saving, which would imply that if the save fails for some other reason, the file would just sit there and never be deleted.

[bemoody]

It seems to work the way I want if I wrap the body of edit_training in transaction.atomic, or if I wrap TrainingForm.save in transaction.atomic, or even if I just wrap training.save in transaction.atomic. I'm going with "autocommit is inherently screwy".

TrainingForm.save should be atomic anyway.

Still, if raising an exception during an autocommitted thingy can cause breakage outside the scope of the current requuest, that's a problem. And storage backends raising exceptions when you try to save a file is a Thing That Happens. And our error pages shouldn't be hitting the database.