heap buffer overflow is found in upng.c

[iwashiira]

Our fuzzer found heap buffer overflow in upng.c in the current main(1e0953f).

Following is an output of valgrind.
vuln26-patch.png is in vuln26-patch.zip

==39424== Memcheck, a memory error detector
==39424== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==39424== Using Valgrind-3.18.1 and LibVEX; rerun with -h for copyright info
==39424== Command: ./build/bin/lws-api-test-upng --stdin ../crash/vuln26-patch.png
==39424==
[2024/05/15 12:23:14:0930] U: LWS UPNG test tool
==39424== Invalid write of size 1
==39424==    at 0x1193DC: unfilter_scanline (upng.c:209)
==39424==    by 0x11977E: lws_upng_emit_next_line (upng.c:269)
==39424==    by 0x113575: main (main.c:97)
==39424==  Address 0x4f85260 is 0 bytes after a block of size 33,280 alloc'd
==39424==    at 0x48487A9: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==39424==    by 0x1645DD: _realloc (alloc.c:151)
==39424==    by 0x16467F: lws_realloc (alloc.c:207)
==39424==    by 0x11A1A3: lws_upng_decode (upng.c:496)
==39424==    by 0x1195B7: lws_upng_emit_next_line (upng.c:249)
==39424==    by 0x113575: main (main.c:97)
==39424==
[2024/05/15 12:23:14:1709] N: main: wrote 0
[2024/05/15 12:23:14:1732] E: unfilter_scanline: line start is broken 125
[2024/05/15 12:23:14:1738] E: main: emit returned FATAL 13
[2024/05/15 12:23:14:1752] U: Completed: FAIL
==39424==
==39424== HEAP SUMMARY:
==39424==     in use at exit: 0 bytes in 0 blocks
==39424==   total heap usage: 7 allocs, 7 frees, 41,084 bytes allocated
==39424==
==39424== All heap blocks were freed -- no leaks are possible
==39424==
==39424== For lists of detected and suppressed errors, rerun with: -s
==39424== ERROR SUMMARY: 8 errors from 1 contexts (suppressed: 0 from 0)

libwebsockets/lib/misc/upng.c

Lines 464 to 465 in 1e0953f

 u->u.bypp	= (u->u.bpp + 7) / 8; 
 u->inf.bypl = u->u.bypl	= u->width * u->u.bypp; 

The code is built on the assumption that bypl > bypp, but by specifying width = 0, we can create a situation where bypp < bypl.

libwebsockets/lib/misc/upng.c

Line 485 in 1e0953f

 size_t ims = (u->u.bypl * 2) + u->inf.info_size; 

libwebsockets/lib/misc/upng.c

Line 496 in 1e0953f

 u->inf.out = (uint8_t *)lws_malloc(ims, __func__); 

While the size of the heap area is specified using bypl as above, some writes are done based on bypp, resulting in a heap BOF.

For example, in the following code flow.

libwebsockets/lib/misc/upng.c

Line 503 in 1e0953f

 u->u.lines = u->inf.out + u->inf.info_size; 

libwebsockets/lib/misc/upng.c

Lines 262 to 264 in 1e0953f

 obp		= uf->alt ? uf->bypl : 0; 
 uf->precon	= uf->alt ? uf->lines : uf->lines + uf->bypl; 
 uf->recon	= &uf->lines[obp]; 

libwebsockets/lib/misc/upng.c

Lines 208 to 210 in 1e0953f

 for (i = 0; i < uf->bypp; i++) 
 	uf->recon[i] = (uint8_t)(u->inf.out[(uf->sp + i) % 
 	                            u->inf.info_size]); 

OOB happens with the writing of uf->recon[i]

Also, the PR of #3134 was found to be inadequate as a check for interger overflow.
For example, when bypl * 2 causes interger overflow, heap BOF occurs as when width = 0

The bypp check is in the determine_format function, so we need a check for width and a more rigorous check for integer overflow in the ims calculation.

libwebsockets/lib/misc/upng.c

Lines 303 to 304 in 1e0953f

 static lws_upng_format_t 
 determine_format(lws_upng_t* upng) { 

Ricerca Security, Inc.