nodemcu / nodemcu-firmware

Lua based interactive firmware for ESP8266, ESP8285 and ESP32

Home Page:https://nodemcu.readthedocs.io

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

RFC Arbitrary files in LFS

vsky279 opened this issue · comments

Missing feature

Support for arbitrary files stored in LFS

Justification

For current application I'd like to have SPIFFS file system free for logs or configuration files. While Lua code is stored in LFS, init.lua file can be there too, there is no easily available solution for an arbitrary files to be stored in LFS now. Such file can be something like index.html, favicon.ico, etc.

I had a discussion with @TerryE Terry who throw me some ideas. I'd like to hear your comments to the proposed solution.

In the linked gist there are several files:

  • file_lfs.lua - Lua module built above standard file module providing transparent access to files stored in LFS and SPIFFS.
  • make_resource.lua - Lua script to be used on PC to generate resource.lua file from files to be embedded in LFS.
  • resource.lua - example of favicon.ico file to be included in LFS

Usage

The module is intended to replace standard file module. If file is present in SPIFFS then this version is preferred. If it is not present then read only access (of course) to the file in LFS is provided.
I see big advantage of this solution that no change to the existing code needs to be done. Performance of file function seems to be affected though this does not seem to be critical for most applications (no benchmarking has been done).

The following functions for LFS files are supported:

  • file.exists()
  • file.open
  • file.read(), file.obj:read()
  • file.readline(), file.obj:readline()
  • file.seek(), file.obj:seek()

Other functions are just passed-through to the standard file module.

When file is embedded in LFS, no standard userdata file structure is created. Instead a Lua object is created.

Example

file = require("file_lfs")

local idx = file.open("index.html") -- file not in SPIFFS but in resource.lua in LFS
print(idx:readline())
idx:seek("set", 0)
print(file.readline(idx))

idx = file.open("favicon.ico", "r") -- file not in SPIFFS but in resource.lua in LFS
print(file.read(512))
file.seek(idx, "cur", 100)
print(idx:read(16))
idx:close()

local i = file.open("init.lua") -- files in SPIFFS are accessible using same routines
print(i:readline())

Looks good to me.
There are some things to clean out I think but we should discuss that when you made a PR.
I also had a PR about this a while ago but implemented directly in luac.cross which was much disliked by Terry.
But your Integration in the file module is even a step further.

I would like to see an errormessage for write access on an LFS file. Also a better separation in the non object version would be good.

The check on every cll to seek, read ... for lfs or real file does not seem to be needed in the object version and should not be there. Just takes up time.

From reading the soueces I think that also the esists needs tuning and should call the file base function if not in LFS. or maybe give the priority to the file system.

We introduced a new test framework some days ago. Maybe you want to write some tests which would also ease your development if you have tests in place that you can run in seconds.

Gregor, thanks for your comments.

I would like to see an errormessage for write access on an LFS file. Also a better separation in the non object version would be good.

The idea was that you can still write to a file that has the same name as a file in LFS. It results in a new file in SPIFFS (that is then preferred when opened again).

f=file.open("index.html")
print(f:readline())
-- prints: <!DOCTYPE html><html><head>...

f=file.open("index.html","w")
f:write("test")
f:close()

f=file.open("index.html")
print(f:readline())
-- prints: test

The check on every call to seek, read ... for lfs or real file does not seem to be needed in the object version and should not be there. Just takes up time.

Well that allows elegant usage such as f1=file.open("index.html");f2=file.open("init.lua");print(f1.read(f2, 10)) which is ... utterly nonsense 😃. Good point. gist is updated.

From reading the sorces I think that also the exists needs tuning and should call the file base function if not in LFS. or maybe give the priority to the file system.

return (node_LFS_resource(filename) ~= nil) or file_exists(filename)

I think that this is happening. If the file is not in LFS the file base function is called.

We introduced a new test framework some days ago. Maybe you want to write some tests which would also ease your development if you have tests in place that you can run in seconds.

I'll have a look and I will prepare a PR.

Gregor, thanks for your comments.

I would like to see an errormessage for write access on an LFS file. Also a better separation in the non object version would be good.

The idea was that you can still write to a file that has the same name as a file in LFS. It results in a new file in SPIFFS (that is then preferred when opened again).

I see. That works great when you use 'w' mode to open it but yields unexpected results if you use one of the modes which only modify parts of the file contents. In these cases the file could be copied to SPIFFS completely before proceeding. That would be a transparent experience. So we should have at least meaningfull errors there I think.

The check on every call to seek, read ... for lfs or real file does not seem to be needed in the object version and should not be there. Just takes up time.

Well that allows elegant usage such as f1=file.open("index.html");f2=file.open("init.lua");print(f1.read(f2, 10)) which is ... utterly nonsense 😃. Good point. gist is updated.

Uhh had to read this twice. Nice code!

From reading the sorces I think that also the exists needs tuning and should call the file base function if not in LFS. or maybe give the priority to the file system.

return (node_LFS_resource(filename) ~= nil) or file_exists(filename)

I think that this is happening. If the file is not in LFS the file base function is called.

I have been viewing it on my phone. must have missed the part of the line which scrolled out of view ...

You will also want to cover these functions

file.getcontents()
file.list()
file.rename()
file.stat()

I think handling the non object model also needs some love.

Imagine

file.open("index.html")
print(file.readline())

would call the LFS file method for open and then the original file method for readline if I am not mistaken.

closing as the PR is merged already for a while now