potrace: memory allocation failure

[Agostino Sarubbo <ago () gentoo org>]

Description:
potrace is a utility that transforms bitmaps into vector graphics.

A crafted image, through a fuzz testing, causes the memory allocation to fail.

This is the first case where my ASan symbolyzer didn’t start up correctly. I’m 
reporting only what it prints at the end (not useful at all but demostrates a 
bit that the issue exist)

# potrace $FILE
potrace: warning: 2.hangs: premature end of file
==13660==ERROR: AddressSanitizer failed to allocate 0x200003000 (8589946880) 
bytes of LargeMmapAllocator (error code: 12)
==13660==AddressSanitizer CHECK failed: /var/tmp/portage/sys-
devel/llvm-3.8.1/work/llvm-3.8.1.src/projects/compiler-
rt/lib/sanitizer_common/sanitizer_common.cc:183 "((0 && "unable to mmap")) != 
(0)" (0x0, 0x0)

Affected version:
1.13

Fixed version:
N/A

Commit fix:
N/A

Credit:
This bug was discovered by Agostino Sarubbo of Gentoo.

CVE:
N/A

Timeline:
2016-08-26: bug discovered
2016-08-27: bug reported privately to upstream
2016-08-29: blog post about the issue

Note:
This bug was found with American Fuzzy Lop.

Permalink:
https://blogs.gentoo.org/ago/2016/08/29/potrace-memory-allocation-failure/