jasper: multiple assertion failures

[Agostino Sarubbo <ago () gentoo org>]

If it is suitable for a CVE please assign one. Thanks.

Description:
jasper is an open-source initiative to provide a free software-based reference 
implementation of the codec specified in the JPEG-2000 Part-1 standard.

A fuzzing revealed multiple assertion failures.
Since the jasper’s maintainer releases frequently, the fuzzing was done across 
multiple versions. The “affected version” tag means that it was tested and 
discovered on that version, so previously versions may be affected too.
The latest failures are unfixed. I will update the post when upstream will 
work on them.

Affected version:
1.900.12
Output/failure:
imginfo: /tmp/portage/media-
libs/jasper-1.900.12/work/jasper-1.900.12/src/libjasper/base/jas_seq.c:90: 
jas_matrix<= yend' failed.
Commit fix:
https://github.com/mdadams/jasper/commit/d91198abd00fc435a397fe6bad906a4c1748e9cf
Fixed version:
1.900.13
Testcase:
https://github.com/asarubbo/poc/blob/master/00003-jasper-assert-jas_matrix_t

######################################################

Affected version:
1.900.13
Output/failure:
/tmp/portage/media-
libs/jasper-1.900.13/work/jasper-1.900.13/src/libjasper/ras/ras_dec.c:330: int 
ras_getcmap(jas_stream_t *, ras_hdr_t *, ras_cmap_t *): Assertion `numcolors 
<= 256' failed.
Commit fix:
https://github.com/mdadams/jasper/commit/411a4068f8c464e883358bf403a3e25158863823
Fixed version:
1.900.14
Testcase:
https://github.com/asarubbo/poc/blob/master/00005-jasper-assert-ras_getcmap

######################################################

Affected version:
1.900.13
Output/failure:
imginfo: /tmp/portage/media-
libs/jasper-1.900.13/work/jasper-1.900.13/src/libjasper/jpc/jpc_mct.c:146: 
void jpc_irct(jas_matrix_t *, jas_matrix_t *, jas_matrix_t *): Assertion 
`((c1)->numrows_) == numrows && ((c1)->numcols_) == numcols && ((c2)-
> numrows_) == numrows && ((c2)->numcols_) == numcols’ failed.
Commit fix:
https://github.com/mdadams/jasper/commit/dee11ec440d7908d1daf69f40a3324b27cf213ba
Fixed version:
1.900.14
Testcase:
https://github.com/asarubbo/poc/blob/master/00006-jasper-assert-jpc_irct

######################################################

Affected version:
1.900.13
Output/failure:
type = 0xff76 (UNKNOWN); len = 20;10 40 40 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 imginfo: /tmp/portage/media-
libs/jasper-1.900.13/work/jasper-1.900.13/src/libjasper/jpc/jpc_mct.c:233: 
void jpc_iict(jas_matrix_t *, jas_matrix_t *, jas_matrix_t *): Assertion 
`((c1)->numcols_) == numcols && ((c2)->numcols_) == numcols’ failed.
Commit fix:
https://github.com/mdadams/jasper/commit/dee11ec440d7908d1daf69f40a3324b27cf213ba
Fixed version:
1.900.14
Testcase:
https://github.com/asarubbo/poc/blob/master/00008-jasper-assert-jpc_iict

######################################################

Affected version:
1.900.13
Output/failure:
imginfo: /tmp/portage/media-
libs/jasper-1.900.13/work/jasper-1.900.13/src/libjasper/base/jas_seq.c:90: 
jas_matrix_t *jas_seq2d_create(int, int, int, int): Assertion `xstart <= xend 
&& ystart <= yend' failed.
Commit fix:
https://github.com/mdadams/jasper/commit/ba2b9d000660313af7b692542afbd374c5685865
Fixed version:
1.900.14
Testcase:
https://github.com/asarubbo/poc/blob/master/00007-jasper-assert-jas_matrix_t

######################################################

Affected version:
1.900.13
Output/failure:
type = 0xff05 (UNKNOWN); len = 20;01 40 40 00 f0 00 00 00 00 00 00 00 00 00 00 
00 00 00 imginfo: /tmp/portage/media-
libs/jasper-1.900.13/work/jasper-1.900.13/src/libjasper/jpc/jpc_bs.c:197: long 
jpc_bitstream_getbits(jpc_bitstream_t *, int): Assertion `n >= 0 && n < 32' 
failed.
Commit fix:
https://github.com/mdadams/jasper/commit/1e84674d95353c64e5c4c0e7232ae86fd6ea813b
Fixed version:
1.900.14
Testcase:
https://github.com/asarubbo/poc/blob/master/00014-jasper-assert-jpc_bitstream_getbits

######################################################

Affected version:
1.900.13
Output/failure:
imginfo: /tmp/portage/media-
libs/jasper-1.900.13/work/jasper-1.900.13/src/libjasper/jpc/jpc_dec.c:1637: 
void calcstepsizes(uint_fast16_t, int, uint_fast16_t *): Assertion `!((expn + 
(numrlvls – 1) – (numrlvls – 1 – ((bandno > 0) ? ((bandno + 2) / 3) : (0)))) & 
(~0x1f))’ failed.
Commit fix:
https://github.com/mdadams/jasper/commit/f7038068550fba0e41e1d0c355787f1dcd5bf330
Fixed version:
1.900.17
Testcase:
https://github.com/asarubbo/poc/blob/master/00012-jasper-assert-calcstepsizes

######################################################

Affected version:
1.900.13
Output/failure:
type = 0xff41 (UNKNOWN); len = 20;02 40 40 00 00 00 00 ee ff 00 00 00 00 24 00 
00 00 00 imginfo: /tmp/portage/media-
libs/jasper-1.900.13/work/jasper-1.900.13/src/libjasper/jpc/jpc_t2cod.c:297: 
int jpc_pi_nextrpcl(jpc_pi_t *): Assertion `pi->prcno pirlvl->numprcs’ failed.
Commit fix:
https://github.com/mdadams/jasper/commit/f7038068550fba0e41e1d0c355787f1dcd5bf330
Fixed version:
1.900.17
Testcase:
https://github.com/asarubbo/poc/blob/master/00013-jasper-assert-jpc_pi_nextrpcl

######################################################

Affected version:
1.900.15
Output/failure:
imginfo: /tmp/portage/media-
libs/jasper-1.900.15/work/jasper-1.900.15/src/libjasper/base/jas_seq.c:90: 
jas_matrix_t *jas_seq2d_create(int, int, int, int): Assertion `xstart <= xend 
&& ystart <= yend' failed.
Commit fix:
https://github.com/mdadams/jasper/commit/f7038068550fba0e41e1d0c355787f1dcd5bf330
Fixed version:
1.900.17
Testcase:
https://github.com/asarubbo/poc/blob/master/00016-jasper-assert-jas_matrix_t

######################################################

Affected version:
1.900.22
Output/failure:
warning: trailing garbage in marker segment (9 bytes)
warning: trailing garbage in marker segment (40 bytes)
warning: ignoring unknown marker segment (0xffee)
type = 0xffee (UNKNOWN); len = 23;1f 32 ff ff ff 00 10 00 3d 4d 00 01 32 ff 00 
e4 00 10 00 00 4f warning: trailing garbage in marker segment (34 bytes)
imginfo: /tmp/portage/media-
libs/jasper-1.900.22/work/jasper-1.900.22/src/libjasper/base/jas_seq.c:90: 
jas_matrix_t *jas_seq2d_create(int, int, int, int): Assertion `xstart <= xend 
&& ystart <= yend' failed.
Commit fix:
https://github.com/mdadams/jasper/commit/d42b2388f7f8e0332c846675133acea151fc557a
Fixed version:
1.900.25
Testcase:
https://github.com/asarubbo/poc/blob/master/00043-jasper-assert-jas_matrix_t

######################################################

Affected version:
1.900.13
Output/failure:
/tmp/portage/media-
libs/jasper-1.900.13/work/jasper-1.900.13/src/libjasper/jpc/jpc_t1cod.c:144: 
int JPC_NOMINALGAIN(int, int, int, int): Assertion `qmfbid == 0x01′ failed.
Commit fix:
N/A
Fixed version:
N/A
Testcase:
https://github.com/asarubbo/poc/blob/master/00004-jasper-assert-JPC_NOMINALGAIN

######################################################

Affected version:
1.900.13
Output/failure:
type = 0xff76 (UNKNOWN); len = 20;00 40 40 00 00 00 00 00 00 00 00 00 00 00 00 
00 00 00 imginfo: /tmp/portage/media-
libs/jasper-1.900.13/work/jasper-1.900.13/src/libjasper/jpc/jpc_dec.c:1817: 
void jpc_dequantize(jas_matrix_t *, jpc_fix_t): Assertion `absstepsize >= 0′ 
failed.
Commit fix:
N/A
Fixed version:
N/A
Testcase:
https://github.com/asarubbo/poc/blob/master/00010-jasper-assert-jpc_dequantize

######################################################

Affected version:
1.900.17
Output/failure:
imginfo: /tmp/portage/media-
libs/jasper-1.900.17/work/jasper-1.900.17/src/libjasper/jpc/jpc_math.c:94: int 
jpc_floorlog2(int): Assertion `x > 0′ failed.
Commit fix:
N/A
Fixed version:
N/A
Testcase:
https://github.com/asarubbo/poc/blob/master/00023-jasper-assert-jpc_floorlog2

######################################################

Affected version:
1.900.22
Output/failure:
warning: trailing garbage in marker segment (9 bytes)
warning: trailing garbage in marker segment (28 bytes)
warning: trailing garbage in marker segment (40 bytes)
warning: ignoring unknown marker segment (0xffee)
type = 0xffee (UNKNOWN); len = 23;1f 32 ff ff ff 00 10 00 3d 4d 00 01 32 40 e4 
e4 00 10 00 00 4f warning: trailing garbage in marker segment (12 bytes)
imginfo: /tmp/portage/media-
libs/jasper-1.900.22/work/jasper-1.900.22/src/libjasper/jpc/jpc_dec.c:1650: 
void calcstepsizes(uint_fast16_t, int, uint_fast16_t *): Assertion `!((expn + 
(numrlvls – 1) – (numrlvls – 1 – ((bandno > 0) ? ((bandno + 2) / 3) : (0)))) & 
(~0x1f))’ failed.
Commit fix:
N/A
Fixed version:
N/A
Testcase:
https://github.com/asarubbo/poc/blob/master/00044-jasper-assert-calcstepsizes

Credit:
This bug was discovered by Agostino Sarubbo of Gentoo.

Timeline:
2016-10-23: start to report to upstream the issues
2016-11-16: blog post about the issue

Note:
These bugs were found with American Fuzzy Lop.

Permalink:
https://blogs.gentoo.org/ago/2016/11/16/jasper-multiple-assertion-failure