Decoding of metadata is usually cached, using this lazy decoder; for instance here.
But top-level metadata is not cached. So, anytime you do ts.metadata (or tables.metadata) it decodes it anew. This is not ideal in the usual situation because if there's, say, a loop over all mutations, and in the loop someone pulls something out of top-level metadata then the JSON-encoded is parsed every time. This could be fixed by for instance assigning md = ts.metadata and using md in the loop, but this isn't intuitive because ts.metadata looks just like a dictionary.
However, the reason I'm writing this is that with SLiM wanting to store large amounts of data in top-level metadata (using the json+struct codec). So, every time anything is pulled out of top-level metadata, it decodes the entire thing. This happens even if you just want to pull a single number out of the JSON blob (say, ts.metadata["SLiM"]["tick"]) then the whole binary struct blob is decoded also.
We can see the effects of this: here's timing (using python -m timeit) of (a) tskit.load; (b) accessing ts.metadata["SLiM"]["tick"], in JSON; (c) accessing ts.metadata["SLiM_mutation_list"][0]["slim_time"] (a single number pulled out of the binary blob) as a function of the number of mutations:
Runtime to access a single value in metadata is increasing with the number of mutations - which is the size of the binary blob in metadata. Furthermore, it's increasing fast enough it'll be a problem. For instance, with 3M mutations, extrapolating, it'd take 10s to load the metadata (once). Furthermore, since the per-access cost of pulling something out of metadata is proportional to the number of mutations, the cost of looping over mutations and pulling something out of metadata for each is quadratic in the number of mutations.
On the user's side the solution is to do md = ts.metadata and use md as noted above. But this doesn't help out pyslim, for which there are functions (e.g., like ts.slim_time( )) that need to pull something from metadata every time you call the function. We also can't use the existing lazy_decode to wrap the class definition because TreeSequence and TableCollection aren't dataclasses.
(Tangentially, I'm confused about how the usual cacheing works: it seems to me that the usual use case is things like for mut in ts.mutations(); x = mut.metadata['foo'] for which caching would actually slow things down? But I haven't gone back to investigate the original implementation; I think there was benchmarking.)
Decoding of metadata is usually cached, using this lazy decoder; for instance here.
But top-level metadata is not cached. So, anytime you do
ts.metadata(ortables.metadata) it decodes it anew. This is not ideal in the usual situation because if there's, say, a loop over all mutations, and in the loop someone pulls something out of top-level metadata then the JSON-encoded is parsed every time. This could be fixed by for instance assigningmd = ts.metadataand usingmdin the loop, but this isn't intuitive becausets.metadatalooks just like a dictionary.However, the reason I'm writing this is that with SLiM wanting to store large amounts of data in top-level metadata (using the json+struct codec). So, every time anything is pulled out of top-level metadata, it decodes the entire thing. This happens even if you just want to pull a single number out of the JSON blob (say,
ts.metadata["SLiM"]["tick"]) then the whole binary struct blob is decoded also.We can see the effects of this: here's timing (using
python -m timeit) of (a)tskit.load; (b) accessingts.metadata["SLiM"]["tick"], in JSON; (c) accessingts.metadata["SLiM_mutation_list"][0]["slim_time"](a single number pulled out of the binary blob) as a function of the number of mutations:Runtime to access a single value in metadata is increasing with the number of mutations - which is the size of the binary blob in metadata. Furthermore, it's increasing fast enough it'll be a problem. For instance, with 3M mutations, extrapolating, it'd take 10s to load the metadata (once). Furthermore, since the per-access cost of pulling something out of metadata is proportional to the number of mutations, the cost of looping over mutations and pulling something out of metadata for each is quadratic in the number of mutations.
On the user's side the solution is to do
md = ts.metadataand usemdas noted above. But this doesn't help out pyslim, for which there are functions (e.g., likets.slim_time( )) that need to pull something from metadata every time you call the function. We also can't use the existinglazy_decodeto wrap the class definition becauseTreeSequenceandTableCollectionaren't dataclasses.(Tangentially, I'm confused about how the usual cacheing works: it seems to me that the usual use case is things like
for mut in ts.mutations(); x = mut.metadata['foo']for which caching would actually slow things down? But I haven't gone back to investigate the original implementation; I think there was benchmarking.)