Giters

adafruit / Adafruit_CircuitPython_Bundle

A bundle of useful CircuitPython libraries ready to use from the filesystem.

Geek Repo:Geek Repo

Github PK Tool:Github PK Tool

Put cpx, crickit, and other always-frozen libs in another directory to save RAM

dhalbert opened this issue · comments

commented

We recently ran a hacking session with CPX's. People had lib/ directories on their CPX's, because they were using adafruit_irremote. As a result, they were getting the non-frozen version of adafruit_circuitplayground, and were running out of memory. We had to tell them to remove that lib from the lib/ directory, so they could use the frozen versions instead

This is a common support issue. The workaround may already be documented, but it's one of a long list of things to know in the Guides. Suppose we made getting the frozen versions more automatic. There are several ways to do this, but the main thing is to move those libs off sys.path or put them in a directory after .frozen on the path.

  1. There could be a libextra or a libfrozen or libopt directory, which would contain only adafruit_circuitplayground and adafruit_crickit libraries, since they are always frozen in on the boards those specific boards.
  2. We could have multiple .zip files, with some targeted towards particular boards, just like we have multiple .uf2 files. That would allow moving other frozen libs (like NeoPixel) into the secondary `lib directory.
  3. There could be a single master zip with multiple folders (e.g., lib-cpx, lib-cpx-crickit), some of which had the frozen libs removed, and the user would select the right one to copy.

In the list above, I kind of like 2 best, but let's discuss.

Another related idea, which solves a different problem (not this issue) is to include the mpy version in the name, like lib2, lib3, etc. and customize sys.path accordingly in the build. That helps solves the major-version update problem.

tagging:
@kattni (who was at this session)
@tannewt
@ladyada

commented

would this also apply to the pirkey_m0?

commented

@jerryneedell Good point, yes. Since we're not selling it currently, I wasn't thinking about that.

commented

oh yeah thats a good idea
put /libextra first, .frozen second, then /lib ?

commented

@ladyada other way around: most libraries in lib, then .frozen, then libextra which will be only the ones that are usually frozen, especially cpx and crickit.

commented

i meant the searchpath :)

commented

There are two issues being covered here, and I agree that both need to be addressed. I'm creating a second issue with the second topic so we can focus on each on accordingly.

The least invasive way to do this is to change sys.path. It would all be handled behind the scenes at that point. It seems like if we're going to do it, we should put it as ['', '/', '.frozen', '/lib']. That leaves us a way to test updated libraries without needing to do a build with them frozen in every time. In situations where people are dragging individual libraries, they're often putting them on CIRCUITPY not in /lib, so we're still going to have to address that as it happens.

Having multiple zip files for bundles is also a good option. We have 5 boards with frozen libraries, and would need to maintain the current two main bundle versions, which comes to 7 total (I think). We've already introduced people to the idea of downloading something specific to their board with CircuitPython, so this isn't a new concept. This would be a more invasive way to resolve this issue because we're now introducing 5 more options into the bundle download situation and will obviously run into the issue of people needing to update their bundle and people having the wrong bundle loaded.

I think moving some of the libraries to a different folder is the least useful way to do it as it can't reasonably address all the libraries frozen into the various builds without running into issues on boards that don't have libs frozen in.

Regardless of how we do this people will have to update something: either the firmware or the lib bundle. So the decision comes down to where to we want to make the change and what thing do we want people to have to update.

commented

What if we took a step back and tried to solve the original problem of memory use?

Perhaps the best thing is to use spare flash space to host libraries pulled from flash. Or, better yet, leave the raw code bits from the mpy file on disk until they are actually used. This will make better use of ram for all cases instead of the specific cases that frozen modules covers. It'll also leave the bundle as is.

commented

I agree with Kattni's thoughts that simply flipping .frozen and lib in the search path is the easiest solution. A long time ago we talked about putting .frozen first, but were then worried about not being able to update frozen libraries without a firmware update. At that time the libraries were much less mature. Now they are in good shape. But we see support cases where people run out of RAM because they have duplicate copies of the the frozen libs in their lib directory.

I think it's a nice aspirational idea to have an "mpy cache" in flash, which would get updated dynamically if a newer mpy was available in the filesystem on import. But that's a significant implementation task.

We can talk about this in the meetings today and later too.

commented

My concern with having .frozen first by default is that is then becomes more difficult to use an updated version of a library. I suppose we could just test by putting the test candidate at the root level instead of in /lib then it would get used before .frozen.

I am a little uncomfortable intentionally having two version of a library installed - one in .frozen and one in /lib. For my workflow, I just remove the redundant entries from /lib then the current search path is fine.

It may be simpler to hide this by putting .frozen first and that is will work a lot of users. It really depends on how much we want to educate the user to the internal workings.

commented

If a module exists as frozen then I think there has already been an implicit decision that it should be used first over whatever is in the lib folder. Further, the situation of having a frozen module but wanting to use a non-frozen version instead is kind of an advanced technique. How about this?

Put .frozen first so this would import frozen version:

import foo

If you want to get around that, use a different import syntax to import version in lib folder:

import lib.foo as foo
commented

@caternuson makes a good point - as did @kattni - using a non-default version of the library is an advanced technique - so it makes sense for it to be more complex.

I agree that swapping /lib and .frozen is the best way to go.

That said, I still think we should consider distributing different .zip files for each board so we are not intentionally having two versions of the library present.

commented

Another idea - just remove always frozen modules from the bundle.

  • CPX - it's frozen for the only board that uses it
  • crickit - frozen in specific firmware
commented

Not sure about crickit - since we don't produce images for all the feathers that can be used with the feather crickit. For example feather_m4_express. Will it work with any of the basic feather M0's? I have not tried those. What about the feather ESP8266?

commented

crickit and seesaw are not always frozen, and if you're using ESP8266 or M4 yeah you'd install that by hand

commented

I don't like having special cases for crickit and cpx. They have a long term cost during development because they require special treatment forever more. For example, the networking PR requires extra effort because the new code fits in all express boards except those with frozen libraries.

There are a number of ideas for memory optimization that would help all of our users here. Reducing memory use would remove the need to have these libraries frozen in at all, simplify the download options for CircuitPython and simplify development and maintenance.

commented

@tannewt percentage-wise, how much space does dropping the line numbers in .mpy gain? I'm worried it will make support harder, since we won't be able to tell where library code is breaking when presented with backtraces.

commented

I understand the sentiment of not having special cases. But with only 20kB for the heap on the M0, having frozen modules makes an enormous amount of difference toward being able to write programs of significant size on the CPX and Crickit, versus only being able to write toy programs. The CPX library is about 8kB, the LIS3DH library is about 6kB, and the crickit library is about 6kB. These libraries have already been tuned to be smaller. Completely removing these large chunks of .mpy from RAM makes a huge difference. 1kB here and there will not solve the problem.

The M4 and nRF52840 solve the problem in the long run, but there's no large-RAM CPX yet, and we still need to support the large number of boards already sold. At some point I think we should stop adding features to the M0 builds, and save those features for the large-memory boards.

One way to think about the frozen modules is not that they are Python modules that deserve to be compilable and be in RAM, but that they are specialized builtin modules that happen to be coded in Python instead of C. We could have written the libraries above in C to put them in flash, but it's a much more arduous undertaking.

My own personal experience has been trying to write a relatively simple but "real" HID keyboard/mouse program that also supported a character LCD display on the M0. I did all kinds of pruning on the libraries, but without frozen modules, it was impossible.

commented

@dhalbert The behavior without line numbers is that the stack trace says line 1 of the file instead of the actual line. This would require us to reproduce the issue when using the py file but I don't think that prevents debugging.

I ran some approximate numbers yesterday by measuring CPX build sizes with libraries turned on and off and with line number info or not. The number is the number of bytes free in flash. Its a decent approximation of ram size I believe as well.

6928 w/o lines
5484 w/ lines
29372 w/o frozen libs
27440 w/bus device
24592 w/circuitplayground
20100 w/HID
24632 w/lis3dh
26456 w/neopixel
28456 w/thermistor

So, the frozen libs are 23888 bytes and line number info is 1444 bytes (6%). That won't solve our problems but it would help.

There are a couple of other issues with frozen modules:

  • Upgrading them requires a new version of CircuitPython. This is currently coming up in my discussions with the KMK keyboard folks (KMKfw/kmk_firmware#52). They want to freeze their code in but then they must rerelease CircuitPython for every version of their code. Instead, it'd be awesome for them to use stock CircuitPython but then have an easy install/upgrade on top of it.
  • Freezing libs is too easy of an answer when running out of memory and causes design issues to be ignored. For example, the Seesaw library is 10696 bytes (when frozen in)! I doubt it needs to be. This impacts all uses of the Seesaw library but freezing it for Crickit only solves it for those builds.

Ultimately, I think the problem is that we don't have good insight into why we run out of memory. Making it easier for anyone to get a diagram of the heap would help this. We could even automate heap charts for every library.

So, for example, for every bundle we could generate heap charts of all the libraries after import and one for every example. That would make the impact of code changes on memory much more apparent.

Improving library sizes will help all embedded use of them. Changing memory behavior in CircuitPython will as well. Freezing modules doesn't.

commented

This is coming up constantly now. The most reasonable solution is to switch the path to ['', '/', '.frozen', '/lib'] to resolve this. It's not reasonable right now to fix all of the memory issues we're seeing in the M0 builds. There is no quick answer to fixing the underlying issue, and we need something in place now.

I discussed it again with @caternuson because he's a perfect example of the support people who are dealing with this. If we make this change, the thing that will have to happen is for users to upgrade to the latest version of CircuitPython, a step which is much simpler to identify and explain than explaining frozen modules and the reason for deleting the library from the /lib folder.

I will be filing an issue with the same info on the CircuitPython repo as this is a change that needs to be made to CircuitPython, and not to the bundle.