RFC Arbitrary files in LFS
vsky279 opened this issue · comments
Missing feature
Support for arbitrary files stored in LFS
Justification
For current application I'd like to have SPIFFS file system free for logs or configuration files. While Lua code is stored in LFS, init.lua
file can be there too, there is no easily available solution for an arbitrary files to be stored in LFS now. Such file can be something like index.html
, favicon.ico
, etc.
I had a discussion with @TerryE Terry who throw me some ideas. I'd like to hear your comments to the proposed solution.
In the linked gist there are several files:
file_lfs.lua
- Lua module built above standardfile
module providing transparent access to files stored in LFS and SPIFFS.make_resource.lua
- Lua script to be used on PC to generateresource.lua
file from files to be embedded in LFS.resource.lua
- example offavicon.ico
file to be included in LFS
Usage
The module is intended to replace standard file
module. If file is present in SPIFFS then this version is preferred. If it is not present then read only access (of course) to the file in LFS is provided.
I see big advantage of this solution that no change to the existing code needs to be done. Performance of file
function seems to be affected though this does not seem to be critical for most applications (no benchmarking has been done).
The following functions for LFS files are supported:
file.exists()
file.open
file.read()
,file.obj:read()
file.readline()
,file.obj:readline()
file.seek()
,file.obj:seek()
Other functions are just passed-through to the standard file
module.
When file is embedded in LFS, no standard userdata file structure is created. Instead a Lua object is created.
Example
file = require("file_lfs")
local idx = file.open("index.html") -- file not in SPIFFS but in resource.lua in LFS
print(idx:readline())
idx:seek("set", 0)
print(file.readline(idx))
idx = file.open("favicon.ico", "r") -- file not in SPIFFS but in resource.lua in LFS
print(file.read(512))
file.seek(idx, "cur", 100)
print(idx:read(16))
idx:close()
local i = file.open("init.lua") -- files in SPIFFS are accessible using same routines
print(i:readline())
Looks good to me.
There are some things to clean out I think but we should discuss that when you made a PR.
I also had a PR about this a while ago but implemented directly in luac.cross which was much disliked by Terry.
But your Integration in the file module is even a step further.
I would like to see an errormessage for write access on an LFS file. Also a better separation in the non object version would be good.
The check on every cll to seek, read ... for lfs or real file does not seem to be needed in the object version and should not be there. Just takes up time.
From reading the soueces I think that also the esists needs tuning and should call the file base function if not in LFS. or maybe give the priority to the file system.
We introduced a new test framework some days ago. Maybe you want to write some tests which would also ease your development if you have tests in place that you can run in seconds.
Gregor, thanks for your comments.
I would like to see an errormessage for write access on an LFS file. Also a better separation in the non object version would be good.
The idea was that you can still write to a file that has the same name as a file in LFS. It results in a new file in SPIFFS (that is then preferred when opened again).
f=file.open("index.html")
print(f:readline())
-- prints: <!DOCTYPE html><html><head>...
f=file.open("index.html","w")
f:write("test")
f:close()
f=file.open("index.html")
print(f:readline())
-- prints: test
The check on every call to seek, read ... for lfs or real file does not seem to be needed in the object version and should not be there. Just takes up time.
Well that allows elegant usage such as f1=file.open("index.html");f2=file.open("init.lua");print(f1.read(f2, 10))
which is ... utterly nonsense
From reading the sorces I think that also the exists needs tuning and should call the file base function if not in LFS. or maybe give the priority to the file system.
return (node_LFS_resource(filename) ~= nil) or file_exists(filename)
I think that this is happening. If the file is not in LFS the file base function is called.
We introduced a new test framework some days ago. Maybe you want to write some tests which would also ease your development if you have tests in place that you can run in seconds.
I'll have a look and I will prepare a PR.
Gregor, thanks for your comments.
I would like to see an errormessage for write access on an LFS file. Also a better separation in the non object version would be good.
The idea was that you can still write to a file that has the same name as a file in LFS. It results in a new file in SPIFFS (that is then preferred when opened again).
I see. That works great when you use 'w' mode to open it but yields unexpected results if you use one of the modes which only modify parts of the file contents. In these cases the file could be copied to SPIFFS completely before proceeding. That would be a transparent experience. So we should have at least meaningfull errors there I think.
The check on every call to seek, read ... for lfs or real file does not seem to be needed in the object version and should not be there. Just takes up time.
Well that allows elegant usage such as
f1=file.open("index.html");f2=file.open("init.lua");print(f1.read(f2, 10))
which is ... utterly nonsense😃 . Good point. gist is updated.
Uhh had to read this twice. Nice code!
From reading the sorces I think that also the exists needs tuning and should call the file base function if not in LFS. or maybe give the priority to the file system.
return (node_LFS_resource(filename) ~= nil) or file_exists(filename)I think that this is happening. If the file is not in LFS the file base function is called.
I have been viewing it on my phone. must have missed the part of the line which scrolled out of view ...
You will also want to cover these functions
file.getcontents()
file.list()
file.rename()
file.stat()
I think handling the non object model also needs some love.
Imagine
file.open("index.html")
print(file.readline())
would call the LFS file method for open
and then the original file method for readline
if I am not mistaken.
closing as the PR is merged already for a while now