This repository was archived by the owner on Jan 27, 2019. It is now read-only.
Open
Conversation
added 5 commits
February 9, 2017 10:57
The values we store in the .expand_caches are immutable (if we make a change to one of the vars in the dependency set, the entry gets deleted). This means we may share those tuples among all the metadata instances, saving a significant amount of memory. First, this requires changing the second member to a frozenset, and that reveals a (harmless) bug in the override handling; one shouldn't mutate the dependency set one gets from a _get call (the effect was just to add "OVERRIDES" to the dependency set for the "OVERRIDES" variable itself). If we have other such bugs hiding somewhere, the change to using a frozenset should flush them out as well. Then it's just a matter of creating two dicts for holding the interned frozensets and tuples, using the same strategy as Python uses internally for interning strings. As a bonus, we get to drop the optimization where we used None instead of an empty set, since we now use the singleton empty frozenset. The saving is around 7% (40M for oe bake world), with no measurable effect on initialization time. I've done the usual dump of the metadata before and after to check that this does not affect hashes. The dict holding the interned tuples ends up growing to a few MB, so clear the caches after we've done all the hash computations. That may or may not reduce our memory footprint by those MB, depending on the details of Python's and glibc's memory allocation strategy.
I was wondering how we ended up creating ~5000 copies of the strings "machine" and "arm-926ejs-linux-gnueabi" (each thus costing about 3M of memory). It turns out these are the .type and .arch members of the OElitePackage instances, and the reason they each have their own copy is that the string objects are fresh out of the sqlite database. Let's ask pysqlite to intern every string it passes back to us. This saves a lot of memory, and is virtually free.
Every time we do the '(provides or "").split()', python obviously creates a new set of string objects. We stash the strings returned in the __provides flag for a lot of DEPENDS_* and RDEPENDS_* variables, which means we end up having thousands of copies of strings like "libc-6", "libpthread" etc. etc. Interning the strings before passing them back ends up saving more than 50000 string objects, or about 3M of memory.
We cache the compiled actual python functions in the global pythonfunc_code_cache; the PythonFunction wrappers are not and AFAICT cannot be cached (since they have mutable state preventing reuse). So delete the unused attribute, saving about 1M for an oe bake world (280 bytes * 4500 MetaData instances).
The memory footprint of an ExpansionStack instance is 72 bytes for the instance itself, 280 bytes for the __dict__ attribute, and then the memory for the actual list stored at the .stack attribute. We can get rid of the first two contributions by simply letting an ExpansionStack be a list with one extra method that does the sanity checking. Conveniently, the builtin list type even already has a .pop method which does the right thing. Also, __init__ and __len__ are already taken care of. Nobody calls __str__ directly allowing them to pass the optional argument and str(foo, "something else") would be a runtime error, so just make __str__ have the usual prototype. The .python attribute never seems to have been used for anything. Saving 352 bytes per instance adds up to about 1.5M for an oe bake world.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.