Realized that we use zlib API incorrectly throughout the system; not just we assume we can set up the stream with input or output of more than 4GB, we also assume that we can pass the z_stream object around and expect it to keep track of the progress, which is definitely not true if our buffer is larger than what can be expressed with avail_in/avail_out (limited to 4GB).
Scrapped the approach of yesterday's patch series, and tackled the issue with a different approach, namely, to write a thin wrapper layer on top of zlib API to pretend that the API can accept buffers that are larger than zlib supports by making separate and repeated calls into the underlying inflate/deflate as necessary. The trickiest part was not to give Z_FINISH to earlier rounds of inflate/deflate when the buffer is large and the wrapper is feeding the underlying library in separate rounds, which was a stupid bug in my earlier attempts that took quite a long time for me to figure out what is going wrong.
Sent it out as a 7-patch series, and queued them myself.
Scrapped the approach of yesterday's patch series, and tackled the issue with a different approach, namely, to write a thin wrapper layer on top of zlib API to pretend that the API can accept buffers that are larger than zlib supports by making separate and repeated calls into the underlying inflate/deflate as necessary. The trickiest part was not to give Z_FINISH to earlier rounds of inflate/deflate when the buffer is large and the wrapper is feeding the underlying library in separate rounds, which was a stupid bug in my earlier attempts that took quite a long time for me to figure out what is going wrong.
Sent it out as a 7-patch series, and queued them myself.
No comments:
Post a Comment