-
Notifications
You must be signed in to change notification settings - Fork 1
Open
Labels
bugSomething isn't workingSomething isn't working
Description
Definition of Done
- Understand of datatypes in parquet and pyarrow work and verify if we are wasting space
- PIXL to update to pyarrow 19
- If we are wasting space, fix this
PIXL core pins to pyarrow 18.1. However, the pa.decimal32 type was only introduced in pyarrow 19.0.
We wish to use the most compact representation possible, but are forced to use decimal128.
It's possible that the way we're using it is equivalent to decimal32 because of the parameters we pass in:
pa.decimal128(9, 4)
However, this is untested, and we may be wasting lots of space.
(Search for "decimal128" in pseudon.py)
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working