Repository navigation
Read FITS 16-bit integers as signed and expose BZERO/BSCALE in im.info - #10312
karandhaodiyal28-hash wants to merge 5 commits into
Conversation
FITS integers are signed big-endian, but BITPIX=16 data was unpacked with the unsigned I;16B rawmode, so negative values wrapped around and unsigned camera data (stored as signed with BZERO=32768) came out with its top bit flipped. Read 16-bit data as signed I;16BS into mode I, matching the TIFF plugin, and extend the gzip decoder the same way. The BZERO/BSCALE scaling keywords are now exposed in im.info instead of being silently dropped; applying the scaling is left to the caller. Fixes python-pillow#10309
for more information, see https://pre-commit.ci
|
I independently tested Minimal reproduction using Astropy to generate the compressed file (tested with Astropy 8.0.2 and NumPy 2.5.4): from io import BytesIO
import numpy as np
from astropy.io import fits
from PIL import Image
values = np.array([[-32768, -1, 0, 1, 32767]], dtype=np.int16)
output = BytesIO()
fits.HDUList([
fits.PrimaryHDU(),
fits.CompImageHDU(values, compression_type="GZIP_1", tile_shape=(1, 5)),
]).writeto(output)
blob = output.getvalue()
with fits.open(BytesIO(blob)) as hdus:
hdus.verify("exception")
print("Astropy:", hdus[1].data.tolist())
with Image.open(BytesIO(blob)) as image:
print("Pillow:", [image.getpixel((x, 0)) for x in range(image.width)])Output: The decompressed payload contains consecutive 2-byte samples, but the new branch still takes For comparison, current main Validation on Linux x86_64 / Python 3.14.7: AI disclosure: this investigation, executed reproductions and comment were prepared with OpenAI Codex. The account owner has not manually reviewed the code. |
The decompressed GZIP_1 payload holds consecutive samples, while the existing fixture uses groups widened to 4 bytes. Distinguish the two by the decompressed buffer length, like astropy does, and read 16-bit samples as signed in both layouts. This also replaces the reversed slices, which read past the start of the buffer for 32-bit data. Fixes the case reported in python-pillow#10312
|
Good catch, thanks for the thorough report and the clean reproduction. You're right about the cause — the decoder assumed the widened 4-byte layout and skipped every second sample. I've pushed a fix that distinguishes the payload widths by the decompressed buffer length, like astropy does, and reads 16-bit samples as signed in both layouts. The signed 16-bit compressed regression test is added too (Tests/images/gzip_1_16bit.fits), and m13_gzip still decodes identically. While testing the other bit depths I also found the old reversed slice read past the start of the buffer for 32-bit data, so that's fixed along the way (8/16/32-bit and float32 GZIP_1 files now match astropy on my end). Could you re-check with your fixture? |
| if value is None: | ||
| continue | ||
| try: | ||
| number = float(value.replace(b"D", b"E")) |
There was a problem hiding this comment.
Could we preserve integer precision when parsing BZERO and BSCALE?
Right now, every value goes through float() before being converted back to an integer. For example, BZERO = 9007199254740993 would end up stored as 9007199254740992 because the original integer isn't exactly representable as a float.
Since the PR leaves image scaling to the caller, losing precision in the metadata could lead to incorrect pixel values when that scaling is applied.
Maybe we could try parsing integer literals with int() first, then fall back to float() for decimal or exponent notation. A regression test with an integer larger than 2**53 would help.
|
Rechecked The remaining s390x failure looks actionable: A minimal candidate is to retain the current buffer and explicitly select its rawmode at the end of rawmode = {"I": "I;32S", "F": "F;32F"}.get(self.mode, self.mode)
self.set_as_raw(
bytes([pixel for row in rows[::-1] for pixel in row]), rawmode
)These rawmodes decode little-endian signed integers and float32 respectively; The patch coverage check is also below its target (88.88% vs 90.14%). Its uncovered additions include the non-16-bit gzip branches, so committing regression cases for 8-bit, 32-bit and float32 gzip would help cover the decoding changes already exercised above. AI disclosure: this investigation, executed checks and comment were prepared with OpenAI Codex. The account owner has not manually reviewed the code. |
|
The test is currently failing in our big endian CI - https://github.com/python-pillow/Pillow/actions/runs/40511307161/job/122397506881?pr=10312#step:6:10111 |
|
I prepared a tested candidate patch for both open items:
Validation on Linux x86_64 / Python 3.12.12: |
The default rawmodes for the I and F modes are native-endian, so the little-endian rows produced for 32-bit data and expanded 16-bit samples were byte-swapped on big-endian hosts. Pass I;32S and F;32F explicitly so the decoder output is host-independent.
|
Thanks for the big-endian report @radarhere, and for testing @laozidabo — the cause was that the default rawmodes for the "I" and "F" modes are native-endian, so the little-endian rows the decoder builds were byte-swapped on big-endian hosts (0x6F000000 = 111 swapped, matching the failure). Pushed a fix that passes I;32S / F;32F explicitly, so the decoder output is now host-independent. m13 still decodes identically and the astropy comparison matrix (8/16/32-bit, float32, consecutive and widened payloads) passes. |
|
Thanks for the update! One additional observation: for 32-bit integer data, Also, my earlier comment about preserving integer precision for |
|
Confirmed both points on Capturing the rawmode passed to So the 32-bit integer path still falls back to native-endian The two large-integer regression tests also still fail on this revision: both I then applied only the rawmode mapping for all No GitHub Actions check runs have appeared for |
FITS integers are signed big-endian, but BITPIX=16 data was unpacked with the unsigned
I;16Brawmode. Negative values wrapped around, and unsigned camera data (stored as signed withBZERO=32768per Section 5.2) came out with its top bit flipped:This implements option 2 from the issue:
I;16BSinto modeI, like the TIFF plugin does for signed 16-bit. The gzip-compressed path (FitsGzipDecoder) is extended the same way — it expands 2-byte samples to the 4-byteIrawmode so both paths agree.BZERO/BSCALEare now exposed inim.info(int when integral, float otherwise, FITSDexponents handled) instead of being silently dropped. Applying the scaling is left to the caller, per FITS Standard 4.0 Section 4.4.2.5.Adds regression tests building the files in memory, following the reproducer from the issue.
Fixes #10309