Saved scripts are corrupted when closing Notepad++ right before power off
d0vgan opened this issue · comments
Today at about 5 am I closed notepad and other programs, then shutdowned pc, now it's turned out that the file is corrupted. There is definitely no fclose lost anywhere? I mean how is that possible?
Originally posted by @afwn90cj93201nixr2e1re in #8 (comment)
A new branch "feature/saved-scripts-nulls" has been created to handle this.
Please try the latest binaries from:
https://sourceforge.net/projects/npp-plugins/files/NppExec/NppExec%20Plugin%20%28dev%29/
@afwn90cj93201nixr2e1re,
could you check the latest dev binaries?
Just to be sure: are you able to reproduce the problem now, with the latest dev NppExec.dll? Or did it go away?
I've never faced files filled with zeroes personally, so it would be good to have your confirmation of whether the latest NppExec.dll changes something or not.
As i said in last part, I can't reproduce. But i suggested to flush instantly before file close, to save results, and using some kinda winapi functions and flags to make it safer, not libc fopen, yeah. But, it's also acceptable, whatever. It's not saved, but at least it's not corrupted.
Please try the NppExec20210111 from https://sourceforge.net/projects/npp-plugins/files/NppExec/NppExec%20Plugin%20%28dev%29/
It now uses CreateFileW+FILE_FLAG_WRITE_THROUGH for both saving scripts and files as well as for logging.
@afwn90cj93201nixr2e1re
So do you confirm that you have the following identical settings (except the path)?
no, right now i don't have any pc near me, as i said i can send some info in few days.
i can confirm only one thing, my settings always are: first 2 checkboxes, backup in every - 20, backup path - Custom dir, verbose backup - custom dir # 2.
NppExec20210114 is another experimental build where FlushFileBuffers is called before CloseHandle.
Just wondering: whether there will be any difference between NppExec20210111 (without FlushFileBuffers) and NppExec20210114 (with FlushFileBuffers)?
I just tested nppexec for crashes and broke my browser... Ahahaha. I had to go to reinstall, it's good that the profiles didn't break.
@donho
Do you remember in one of the issues I wrote about the fact that my old firefox session was accidentally loaded? Someone asked me why I deleted the comment, etc., it was about all the problems with setting the file modification date (by the way, did you do what I asked?) So, when I shutdown my PC, it really does not have time to save... Lol. Problems with writing files to disk, even with waiting 10-15s. before shutting down the OS, apparently exist in any program. This is such a retarded.
So, as i said:
I tested both of them and both of them do not save files sometimes (i mean probably 99%), at least files aren't corrupted.
Steps:
- Open the script editing window.
- Create some script and add data to it, save (only once).
- Start writing something and press ok, and you need 1-10 seconds after pressing the ok button and executing the script for your computer to shut down.
How do you test? Can you describe your steps? @d0vgan
I will not answer in another place yet to figure it out the reason here, because the problems are the same and the c flag helps to simply avoid corruption sometimes, and I don't even know how to repeat that case with 7.9.2, I just created 10 files, saved them, closed the tabs, exited npp, opened it again, opened the files again, one by one, started editing two of them (split mode), pressed save and turned off the computer after last save button press, one of the files was corrupted with nulls, i was not able to repeat this again. @donho @pnedev
Few days ago i just googled with using this win specified buffer flag (from last 2 commits) and found issue on ms repo where's some guy said that this flag doesn't work and he wrote that he had to change some kind of system setting to actually flush the files to disk.
https://github .com/Microso ftDocs/feedback/issues/2747 (spaces)
Maybe could be interesting too, take a look to transactions, maybe little bit deeper (open link and each link from description in that tab):
https://stackoverflow.com/questions/9047634/flushfilebuffers-on-already-written-file
Thanks for the feedback but it is a bit confusing to me.
Notepad++ and NppExec plugin presumably have the same problem but they do save their files separately so we need to look at them separately.
Since Notepad++ v7.9.1 we use the Windows fopen() c flag which is documented by Microsoft to bypass the Windows buffering and write the file directly to disk on save.
So my question is simple - Have you experienced any file corruption with Notepad++ v7.9.1 or v7.9.2 so far?
I suspect the second Notepad++ backup (Backup on save) setting might still be causing some issues because it uses file copy API which is not specified to bypass the write caching BUT if that backup is disabled and you simply save your files from Notepad++ they should be written successfully with no problems at all.
To summarize, we have 2 Notepad++ scenarios (since v7.9.1):
- Second Notepad++ backup setting (
Backup on Save) disabled: When files are saved, shutting down, restarting or power loss shouldn't corrupt them anymore - files should be successfully saved. - Second Notepad++ backup setting (
Backup on Save) enabled: This might still be causing problems. If you have experienced such thing please confirm, thanks.
We should really continue Notepad++ save problems discussion in notepad-plus-plus/notepad-plus-plus#6133 because this is different from NppExec scripts saving issues - as I said they do save separately so they should be addressed separately.
@d0vgan ,
Thanks for writing and sorry for discussing Notepad++ saving problems here.
First part was for donho, second one for d0vgan, third for you and d0vgan and donho.
Simple answer is yes. As i said in previous discussion and in third part of my last comment, i described how i got file corruption.
and I don't even know how to repeat that case with 7.9.2, I just created 10 files, saved them, closed the tabs, exited npp, opened it again, opened the files again, one by one, started editing two of them (split mode), pressed save and turned off the computer after last save button press, one of the files was corrupted with nulls, i was not able to repeat this again
My backup settings were posted above, ig, then it's second scenario, correct?
You feel free to move discussion where ever you want, im not against at all, but take a look into: https://github.com/d0vgan/nppexec/commits/feature/saved-scripts-nulls, it was done in same way, flag c, that's why we are still discussing such stuff here because it's same problem and both of you tried to fix it in the same way.
I read all the comments in that discussion, most of them are generally useless from a technical point, suggestions to use the c flag and nothing else.
Thank you for your information.
@donho
Do you remember in one of the issues I wrote about the fact that my old firefox session was accidentally loaded? Someone asked me why I deleted the comment, etc.
Sorry, I don't remember that.
I will not answer in another place yet to figure it out the reason here, because the problems are the same and the c flag helps to simply avoid corruption sometimes
I see your point. But there is another factor for causing this issue in Notepad++ : the thread - 7 seconds (20 s in your case) of period backup is run in a thread.
However, I will consider the solution you suggested with @pnedev to use win32 API CreateFile instead of POSIX routine (fopen...).
I suspect the second Notepad++ backup (Backup on save) setting might still be causing some issues because it uses file copy API which is not specified to bypass the write caching BUT if that backup is disabled and you simply save your files from Notepad++ they should be written successfully with no problems at all.
I don't see why Backup on save could be causing the issues. Could you elaborate this point?
notepad-plus-plus/notepad-plus-plus#7839
Can you ask him is that issue reproducible on last version? Probably it wasn't 'reboot'. I randomly found this post while looking for mine.
Im talking about wrong last edited/created time(sometimes): notepad-plus-plus/notepad-plus-plus#5053 (comment)
P.s this files that i showed on screenshots were saved by verbose backup on save actions.
So, can you build some npp release with get file info log after fully processing creating backup file to let me use it and report what exactly is wrong on my system with last edit/creation timestamps, as i said it is very rare, but im using npp a lot, so, i can reproduce it in a week or two.
As for the main task, I just like @d0vgan don't understand whether we should use a specific win api to save files or not, pnedev should take a deep look into win api flags and google harder to check out is issue still reproducible with these flags or not, should we use 2 flags (FILE_FLAG_WRITE_THROUGH, FILE_FLAG_NO_BUFFERING) or only one (FILE_FLAG_WRITE_THROUGH).
As I advised, perhaps in order not to interfere with anyone, we should create the ability to dynamically choose how files should be saved (if possible, but based on what you say about threads it follows that it is difficult to implement correctly).
Few more links:
https://stackoverflow.com/questions/317801/win32-write-to-file-without-buffering
http://www.databaseforum.info/29/1113984.aspx
https://perspectives.mvdirona.com/2008/04/disks-lies-and-damn-disks/
https://www.spinics.net/lists/linux-c-programming/msg00120.html
This is another issue to me, and I cannot reproduce it at all.
Can you ask him is that issue reproducible on last version? Probably it wasn't 'reboot'. I randomly found this post while looking for mine.
According debug info, it's Notepad++ v7.8.2 (64-bit). And I don't think it's reproducible for him steadily either.
So, can you build some npp release with get file info log after fully processing creating backup file to let me use it and report what exactly is wrong on my system with last edit/creation timestamps, as i said it is very rare, but im using npp a lot, so, i can reproduce it in a week or two.
I can build the special release for testing at least. But let's focus on logging "periodic backup" only. notepad-plus-plus/notepad-plus-plus#7839 is another issue which has no correlation with NULL chars issue to me.
So, can you build some npp release with get file info log after fully processing creating backup file to let me use it and report what exactly is wrong on my system with last edit/creation timestamps, as i said it is very rare, but im using npp a lot, so, i can reproduce it in a week or two.
I can build the special release for testing at least.
Yes, i meant only for verbose backup files creation/modified timestamps, because if im gonna ask for that in main repo some random ppls, contributors, indians etc. gonna jump on topic and talk some useless unrelated stuff like: 'i can't repeaaaat and shiiiieeeet, you are stooopit and shiiiieeeet, problem is not related to notepad and shiiieeeet, you should use last version and shiiiieeeet' etc., it's so anoying, i tired after few attempts and left that's why im asking here.
And as i said 7839 probably not relevant, ig it probably was related to same things, but ok.
Hi all,
@d0vgan ,
I think your solution with fopen with c modifier (NppExec commit be31fc1) might not work because in CFileBufT you are not explicitly calling fflush() before closing the file. For reference you can look at https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/fopen-wfopen?view=msvc-160 , I quote the description of c modifier:
c | Enable the commit flag for the associated filename so that the contents of the file buffer are written directly to disk if either fflush or _flushall is called.
Simple answer is yes. As i said in previous discussion and in third part of my last comment, i described how i got file corruption.
Understood, thanks for the feedback.
My backup settings were posted above, ig, then it's second scenario, correct?
Yes, correct. I need to investigate the second backup option (Backup on save) behaviour in detail.
It might take a while, I don't have much spare time recently.
should we use 2 flags (FILE_FLAG_WRITE_THROUGH, FILE_FLAG_NO_BUFFERING) or only one (FILE_FLAG_WRITE_THROUGH)
I think FILE_FLAG_WRITE_THROUGH is enough. You can read the detailed description here https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-createfilea#caching-behavior , but to quote:
If FILE_FLAG_WRITE_THROUGH is used but FILE_FLAG_NO_BUFFERING is not also specified, so that system caching is in effect, then the data is written to the system cache but is flushed to disk without delay.
FILE_FLAG_NO_BUFFERING should be used with caution because if the buffering is not provided by the OS we should make sure our write buffers are properly aligned and sized according to hard-disk geometry (read more here https://docs.microsoft.com/en-us/windows/win32/fileio/file-buffering).
@donho ,
However, I will consider the solution you suggested with @pnedev to use win32 API CreateFile instead of POSIX routine (fopen...).
I personally prefer using the Win32 API for file handling because it gives better programmatic control. That's why my first PR about Notepad++ NUL corruption issue (notepad-plus-plus/notepad-plus-plus#6164) was done using Win32 API.
It is up to you to decide if we should proceed with its re-implementation (I vote 'yes').
However, now that we use fopen with c modifier properly, the NUL corruption shouldn't appear anymore.
At least if we believe the information given by Microsoft.
We must also accept that we have control only to certain extent - you can read details here https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-createfilea#caching-behavior , but let me quote the part I am addressing (it is about the FILE_FLAG_WRITE_THROUGH and FILE_FLAG_NO_BUFFERING Win32 API flags):
Note: Not all hard disk hardware supports this write-through capability.
That is because the flags bypass the Windows caching but the disks themselves might have internal hardware buffers that are not controllable.
I don't see why Backup on save could be causing the issues. Could you elaborate this point?
I suspect that all file manipulation functions use the Windows caching mechanism.
We make sure the OS buffers are flushed to disk when we use fopen and fwrite but APIs like CopyFile, MoveFile, etc. might still be buffered. We need to check where such functions are used in Notepad++ code and if the backup functionality has possible problems related to that.
i meant only for verbose backup files creation/modified timestamps
Do you want me log this option?
What do you want I log creation/modified timestamps ? Just write these 2 timestamps on a log file on each save?
@pnedev
Thank you for your explanation.
I personally prefer using the Win32 API for file handling because it gives better programmatic control. That's why my first PR about Notepad++ NUL corruption issue (notepad-plus-plus/notepad-plus-plus#6164) was done using Win32 API.
It is up to you to decide if we should proceed with its re-implementation (I vote 'yes').
I do understand (and approve) your vision. However, while we are not 100% sure about your last fix (by adding c as argument) not working, I cannot take the risk - not only the amount of change in notepad-plus-plus/notepad-plus-plus#6164 is quite important but also the modified codes are about basic IO operations in Notepad++ - it could bring the regression or side effect, for millions of users.
However, now that we use fopen with c modifier properly, the NUL corruption shouldn't appear anymore.
At least if we believe the information given by Microsoft.
c Enable the commit flag for the associated filename so that the contents of the file buffer are written directly to disk if either fflush or _flushall is called
According the above description, it seems to me c argument makes the effect of FILE_FLAG_WRITE_THROUGH [ FILE_FLAG_NO_BUFFERING, though it's not indicated explicitly.
Note: Not all hard disk hardware supports this write-through capability.
That is because the flags bypass the Windows caching but the disks themselves might have internal hardware buffers that are not controllable.
In this case, even CreateFile with both flags FILE_FLAG_WRITE_THROUGH [ FILE_FLAG_NO_BUFFERING has no effect either.
@donho ,
However, while we are not 100% sure about your last fix (by adding c as argument) not working, I cannot take the risk ...
I understand your reasoning although the chance for regression is nearly zero because no new functionality is added - just the file API is changed. Where and when the files are accessed is not changed, no change in the order of the code execution at all and the new API is not something experimental but the core Windows way to work with files.
Using Win32 API will also fix this issue: notepad-plus-plus/notepad-plus-plus#5664 .
It is currently stalled, I cannot do anything more about it (at least with a reasonable measures that are not overkill).
According the above description, it seems to me
cargument makes the effect ofFILE_FLAG_WRITE_THROUGH
It indeed seems so, so as I said, the NUL corruption on save should have been fixed already and I agree we should focus on examining the Notepad++ backup functionality. All file corruption cases reported recently make use of the backup.
In this case, even CreateFile with both flags FILE_FLAG_WRITE_THROUGH [ FILE_FLAG_NO_BUFFERING has no effect either.
Yes, it seems there might be cases when it will not help but we should do our best and take all measures (that are in our control) to ensure data integrity.
@donho ,
I would suggest to change the following line in Notepad++ Buffer.cpp , bool FileManager::moveFile(BufferID id, const TCHAR * newFileName) function
if (::MoveFileEx(fileNamePath, newFileName, MOVEFILE_REPLACE_EXISTING | MOVEFILE_COPY_ALLOWED) == 0)
to
if (::MoveFileEx(fileNamePath, newFileName, MOVEFILE_REPLACE_EXISTING | MOVEFILE_COPY_ALLOWED | MOVEFILE_WRITE_THROUGH) == 0) .
This is just a precaution when user is renaming files.
For MOVEFILE_WRITE_THROUGH description you can read here:
https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-movefileexa .
For the backup I would suggest the following:
Instead of using the CopyFile function in NppIO.cpp , fileSave() (this line:
if (!::CopyFile(fn, fn_bak.c_str(), FALSE)) )
it seems better (safer) to use the CopyFileEx function (it is documented here https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-copyfileexa) with flag COPY_FILE_NO_BUFFERING .
I don't think it'll change anything because according the screenshot provided by @afwn90cj93201nixr2e1re , the result of "Backup on Save" operation not corrupted (but with the unusual modification time).
It's not harmful either for what you suggested - at least we're sure that the usage of these 2 functions in Notepad++ will avoid system cach.
It's done here:
notepad-plus-plus/notepad-plus-plus@051b17c
I do understand (and approve) your vision. However, while we are not 100% sure about your last fix (by adding c as argument) not working, I cannot take the risk - not only the amount of change in notepad-plus-plus/notepad-plus-plus#6164 is quite important but also the modified codes are about basic IO operations in Notepad++ - it could bring the regression or side effect, for millions of users.
Is it too much work to make a dual solution where the Win32 API would be enabled with a flag? Thus, it could only be gradually implemented and tested by those who are aware of the risks (firstly by those who regularly make copies). When this new approach will not generate any regress and also solve other problems, it could be switch as the main solution.
Is it too much work to make a dual solution where the Win32 API would be enabled with a flag?
I don't see the advantage for the dual solution because:
- It won't change the situation of regression once Notepad++ is released.
- If there's a regression and we need go back to the old implementation,
git revert COMMIT_HASHcan be done for the next release.
I'll go with the (FILE_FLAG_WRITE_THROUGH for CreateFile) plus (FlushFileBuffers before CloseHandle) approach in NppExec. The reasons are:
- these are native Win API features implemented directly by the OS;
- replacing fopen/fread/fwrite with CreateFile/ReadFile/WriteFile removes needless overhead and reduces the binary file size by 13 kB in my case.
Can you build nppexec with write_through + no_buffering for me, just for tests? I mean you can just add the second flag to that branch and compile and post .dll.
From what I see in the documentation, we should either avoid FILE_FLAG_NO_BUFFERING ot use it very-very carefully.
As https://docs.microsoft.com/en-us/windows/win32/fileio/file-buffering mentions:
Because of physical disk layout, file system storage layout, and system-level file pointer position tracking, this write operation will fail unless the locally-defined data buffers meet certain alignment criteria, discussed in the following section.
And the whole "Alignment and File Access Requirements" section of that article (the file-buffering one) describes the requirements.
Given all these details, I don't want to take a risk of using FILE_FLAG_NO_BUFFERING.
Offtop:
donho, you know what's funny? The fact that now i can see when the issue appears, haha.
the first column is creation date, the second one is last edit date.
It is probably related to file saving at all, not only to backup system.
Current version that i'm using.
@afwn90cj93201nixr2e1re
Thank you for the info. I can reproduce it.
I create an issue here:
notepad-plus-plus/notepad-plus-plus#9468
Regarding the "bug" you've observed:
notepad-plus-plus/notepad-plus-plus#9468 (comment)
I've check the code in backup feature in Notepad++, ::CopyFileEx (a Windows native API) is used, not even POSIX compatible fwrite/fopen to copy backup file.
I then googled with "CopyFileEx Date created is later than Date modified", and found the following answer:
That isn't unusual with copied files, when you copy a file the file's created date becomes the modified date and the current date (when the file is copied) becomes the created date, copy any existing file on your computer and you will observe this behaviour.
(ref: https://answers.microsoft.com/en-us/windows/forum/windows_xp-files/how-can-the-date-created-be-later-than-the-date/9e625c0d-8694-4c8e-8a27-3ce74687aa53)
I did Ctrl-C a file created & modified in 2020 in explorer, and Ctrl-V to another folder - the modified date & time of destination file is the same as the source file's created & modified date/time.
Conclusion: it's not a bug. At least not from Notepad++.
In past i suggested to force modified date as current one at least (or add new option to set it as current one, into backup settings) and as you can see on my picture the same file modification have different edited date, but as you said they were copied and edited date should be same. Just take a look onto binding.gyp, at least different files have same random edited date from past, like it is really same, look up to test.js and binding.gyp and addon.cc.
Also how do you think how back up section work? It's definitely make a backup after/before (doesn't matter) saving file, correct? Then how i got random file last edited time (if you are saving some file last edited time obviously should be updated and next time you will get a new one)?
It's definitely a threads issue somehow, not really related to win api in that case.
Can you fix one more thing?
Create folder, create file inside, open that file with notepad, execute some random script inside, for example: echo 123 or NEW LINE cd $(CURRENT_DIRECTORY) NEW LINE ".\asdasdasd.exe" "$(FILE_NAME)";, close the file, close the output window, try to remove folder.
(if you want to use process explorer too, you should close the window after searching because of process explorer cache, it will show you that file is still in use, after reopening that search tab it clears cache, this description is mostly about the next step, unlocking step)
To unlock it you should open for example new file and execute some random script inside it.
I mean you probably understand how you can fix it, you need to close all handles that window close. Also maybe that's related to something else, because sometimes I can't reproduce it, sometimes closing file and window and executing random script in other file directory also doesn't help with releasing.
Also, if you want to create a new issue via using that button, don't use my nickname and screenshots.
You can remove the last part and save the text part, without screenshots.
@afwn90cj93201nixr2e1re
I'd like to ask more questions about this last issue, but it requires a separate discussion. To not mix up very different things here, please create a new issue to proceed.
Folder file view is highly cached, it shouldn't be.
Opening sub-folder cause opening previous folder cache. (As you can see i opened ActionBar but it showed as telegram-ui-actionbar, but should be actionbar.
Also there's no update button and no way to clean cache.
So, 2, 1 cache and no way to update it (i don't see any new folders etc), which probably cause the second, 2 strange folders opening way,
Can you take a look later?
donho.
Can't reproduce after restart.
Any news on this?
I actually faced with another one problem, actually it is probably related to some cache system or something like this, i have opened notepad++ instance which is running about 4-5 days for example.
Let's imagine that i have some folder which opened as workspace. For example telegram source code.
After some events search in files completely ignores some entries.
For example we have file, text file:
abcd asd abcd123
asd123 asd kasd123
asdlasd abcd
abcd
asdasdasdabcd
and if we use Find in Files sometimes it completely ignores some entries, sometimes, i faced with that few times in that month, after reopening some file which exactly contains some entries that weren't found by search and starting search again it shows them too, but only for specific file that was opened in notepad++, not for all missed files.
For example sometimes instead of lines 1, 1, 3, 5, 7 it shows only 1, 7 or 1, 5, like completely randomly.
Same things sometimes happens with Folder as workspace, it ignores new files creations and i can't update list and i need to restart notepad++ just to fix it, after restarting everything works well.
Is there any cache system or something like this in file searches? Why it happens all the time when we use notepad++ more than 3-4 days without restarts? Like real usage, i'm not talking about: Open notepad++, wait 3 days and try it.
I actually described it in previous comment but then i thought that it is probably not related to notepad++ at all and marked it as offtop, but seems like it is related to notepad++ itself.
(https://github.com/notepad-plus-plus/notepad-plus-plus/issues/ 9456#issuecomment-814059483, probably also related to some cache system)
Anyway, i asked for another one unrelated thing in this issue, we probably will hide all unrelated comments later, after resolving, so, there's a few improvements that you can add:
Tabs: Don't switch to previous tab after closing current, for example:
tabs: 1 2 3 4 5 6 7 8 9...
Current is 1, you switched to 100, scrolled whole way.
Current is 100.
You closed 100.
Now you are on 1.
You need to scroll this whole way again to check 99 tab content.
Make a new option for switching switching tabs (after closing current) behaviour.
And the second one related to trailing tabulation, sometimes it is very annoying to remove it after making 2 new lines, one of which should be empty. Can you try to figure out what we can do with that? Because we can't just add option to not set tabulation because of as i showed (gif), sometimes auto tabulation is needed, if column is too far from start, so, i mean, we can't just add option to enabled/disable and mark it as solved. But i can't think of any that can replace it, only ctrl + z to remove tab, but i can't do that in case of 2 new lines, otherwise i need to press tab on second new line to get needed indentation. Any ideas?
@afwn90cj93201nixr2e1re
Could you open an issue regarding this on Notepad++ issues tracker please?
Can you merge those fixes to the main branch asap and release fixed version?
It's already part of released v0.6.2.