Implemented production and reading of zisofs2 for files larger than 4 GiB - 1. New API call iso_stream_get_zisofs_par(). New struct iso_zisofs_ctrl version 2.

doc/zisofs2_format.txt

@@ -0,0 +1,239 @@
+
+                        Description of the zisofs2 Format
+                              Revision 2.0-dev
+
+                           as of zisofs2-tools by
+                   Valentín KIVACHUK BURDÁ and Thomas SCHMITT
+
+                                1 Oct 2020
+
+
+The zisofs2 format was invented by Valentín KIVACHUK BURDÁ and 
+Thomas SCHMITT (as extension of zisofs by H. Peter Anvin). It compresses
+data file content, marks it by a header and provides a pointer array for 
+coarse random access. Within a RRIP enhanced ISO 9660 image the format 
+is additionally marked by a System Use entry with signature "ZF".
+
+The uncompressed size of a single zisofs2 compressed file is restricted
+to 2^64 - 1 bytes. Larger files shall not be compressed. 
+
+The format of version 1 of zisofs is supported by this specification.
+Using it for files with uncompressed size smaller than 4 GiB is friendly
+towards software which does not know about zisofs2.
+See section **LEGACY** for a summary of version 1 of zisofs.
+
+
+
+                                Data Types
+
+ISO 9660:7.3.1 - little endian 4-byte words
+ISO 9660:7.1.1 - unsigned single bytes
+ISO 9660:7.3.3 - 8-bytes value, first in little endian, then big endian.
+#uint64        - 8-bytes unsigned value in little endian
+
+                                Supported compressors
+
+The file header has this layout:
+    @alg_id       @alg_char                Description
+     1            'PZ' (50)(5A)            Zlib
+     2            'XZ' (78)(7A)            XZ
+     3            'L4' (6C)(34)            LZ4
+     4            'ZD' (7A)(64)            Zstandard
+     5            'B2' (62)(32)            Bzip2
+
+@alg_id is a 7.1.1 value. @alg_char is 2 ASCII characters stored as 2 bytes
+Values of @alg_id = 0 and @alg_char = 'pz'(70)(7A) are reserved and 
+must not be used. Other compressors are allowed and may be added to this 
+list in the future
+
+                                Compressor strategy
+
+The default strategy for a compressor is to compress each input data block 
+independently. The zisofs2 spec may define in the future other strategies,
+which will have a new @alg_id, @alg_char and a description in this section.
+
+
+                                File Header
+
+The file header has this layout:
+Offset     Type         Identifier    Contents
+--------------------------------------------------------------------------
+  0        (8 bytes)    @hdr_magic    Magic num (EF 22 55 A1 BC 1B 95 A0)
+  8        7.1.1        @hdr_version  File header version (0)
+  9        7.1.1        @hdr_size     header_size >> 2 (6)
+ 10        7.1.1        @alg_id       Algorithm Type (>=1)
+ 11        7.1.1        @hdr_bsize    log2(block_size) (15, 16, or 17)
+ 12        #uint64      @size         Uncompressed file size
+ 20        (4 bytes)    -             Padding. Ignored
+
+So its size is 24.
+
+Readers shall be able to handle log2(block_size) values 15, 16 and 17
+i.e. block sizes 32 kB, 64 kB, and 128 kB. Writers must not use 
+other sizes.
+
+                               Block Pointers
+
+There are ceil(input_size / block_size) input resp. output blocks.
+Each input block is of fixed size whereas the output blocks have varying
+size (down to 0). For each output block there is an offset pointer giving
+its byte address in the overall file content. The next block pointer in the
+array tells the start of the next block which begins immediately after the
+end of its predecessor. A final pointer (*eob*) gives the first invalid 
+byte address and thus marks the end of the last block.
+
+So there are ceil(input_size / block_size) + 1 block pointers.
+They are stored directly after the file header, i.e. beginning at byte 24, 
+as an array of values in #uint64 format (8 bytes).
+
+Legacy format (zisofs) may be used, which is described in section *LEGACY*
+
+
+
+                                 Data Part
+
+The data part begins immediately after the pointer array (*eob*). In 
+principle it consists of the variable length output blocks as delivered by 
+different compression algorithms when fed with the fixed size input blocks.
+
+A special case of input and output block is defined:
+Zero-length blocks represent a block full of 0-bytes.
+Such input blocks do not get processed by compress2() but shall be mapped 
+to 0-sized output directly. Vice versa 0-sized blocks have to bypass 
+uncompress() when being read.
+
+
+                         ZF System Use Entry Format
+
+The ZF entry follows the general layout of SUSP resp. RRIP.
+Its fields are:
+
+  [1] "BP 1 to BP 2 - Signature Word" shall be (5A)(46) ("ZF").
+
+  [2] "BP 3 - Length" shall specify as an 8-bit number the length in 
+      bytes of the ZF entry recorded according to ISO 9660:7.1.1.
+      This length is 16 decimal.
+      Refer to **LEGACY**
+
+  [3] "BP 4 - System Use Entry Version" shall be 2 as in ISO 9660:7.1.1.
+      Refer to **LEGACY**
+
+  [4] "BP 5 to BP 6 - Algorithm"  shall be two chars to indicate the
+      compression algorithm. For example, (50)(5A) ("PZ")
+      (This is a copy of @alg_char). Refer to **LEGACY**
+
+  [5] "BP 7 - Header Size Div 4" shall specify as an 8-bit number the 
+      number of 4-byte words in the header part of the file data recorded 
+      according to ISO 9660:7.1.1.
+      (This is a copy of @hdr_size).
+
+  [6] "BP 8 - Log2 of Block Size" shall specify as an 8-bit number the 
+      binary logarithm of the compression block size recorded according to
+      ISO 9660:7.1.1.
+      (This is a copy of header byte 13 (@hdr_bsize), resp. header BP 14.
+       The value has to be 15, 16 or 17 i.e. 32 kiB, 64 kiB, or 128 kiB.)
+
+  [7] "BP 9 to BP 16 - Virtual Uncompressed File Size" shall contain
+      as a 64-bit unsigned little endian number the uncompressed 
+      file size represented by the given extent. Refer to **LEGACY**
+       
+
+  | 'Z' | 'F' | LENGTH | 2 | 'P' | 'Z' | HEADER SIZE DIV 4 | 
+  | LOG2 BLOCK SIZE | UNCOMPRESSED SIZE |
+
+Example (block size 128 kiB, uncompressed file size = 40 TB):
+  { 'Z',  'F',  16,   2,  'P',  'Z',    8,   17,
+   0x00, 0x80, 0xCA, 0x39, 0x61, 0x24, 0x00, 0x00 }
+
+
+
+                                 **LEGACY**
+
+zisofs2 supports old readers by respecting the zisofs format. This section
+describes which definitions from zisofs2 must change to be compatible 
+with zisofs.
+
+- General behaviour
+  The uncompressed size of a single zisofs compressed file is restricted
+  to 4 GiB - 1. Larger files shall not be compressed.
+
+- Supported algorithms
+  Only algorithm Zlib with default strategy is supported.
+
+- The file header must follow this structure:
+
+  Offset   Type        Identifier    Contents
+    0      (8 bytes)   @hdr_magic    Magic number (37 E4 53 96 C9 DB D6 07)
+    8      7.3.1       @size         Uncompressed file size
+   12      7.1.1       @hdr_size     header_size >> 2 (4)
+   13      7.1.1       @hdr_bsize    log2(block_size) (15, 16, or 17)
+   14      (2 bytes)   -             Reserved, must be zero
+    
+  So its size is 16.
+
+- Block pointers
+  The array must use ISO 9660:7.3.1 (4 bytes) values.
+
+- ZF entry
+
+Its fields are:
+
+  [1] "BP 1 to BP 2 - Signature Word" shall be (5A)(46) ("ZF").
+
+  [2] "BP 3 - Length" must be 16 decimal.
+
+  [3] "BP 4 - System Use Entry Version" must be 1.
+
+  [4] "BP 5 to BP 6 - Algorithm" must be (70)(7A) ("pz").
+
+  [5] "BP 7 - Header Size Div 4" - same as zisofs2.
+
+  [6] "BP 8 - Log2 of Block Size" - same as zisofs2.
+
+  [7] "BP 9 to BP 16 - Uncompressed Size" This field shall be recorded
+      according to ISO 9660:7.3.3.
+      (This number is the same as @size )
+
+  | 'Z' | 'F' | LENGTH | 1 | 'p' | 'z' | HEADER SIZE DIV 4 | 
+  | LOG2 BLOCK SIZE | UNCOMPRESSED SIZE |
+
+Example (block size 32 kiB, uncompressed file size = 1,234,567 bytes):
+  { 'Z',  'F',   16,    1,  'p',  'z',    4,   15,
+   0x87, 0xD6, 0x12, 0x00, 0x00, 0x12, 0xD6, 0x87 }
+
+
+References:
+
+zisofs2-tools
+  https://github.com/vk496/zisofs2-tools
+
+zisofs-tools
+  http://freshmeat.net/projects/zisofs-tools/
+
+zlib:
+  /usr/include/zlib.h
+
+cdrtools with mkisofs
+  ftp://ftp.berlios.de/pub/cdrecord/alpha
+
+ECMA-119 aka ISO 9660
+  http://www.ecma-international.org/publications/files/ECMA-ST/Ecma-119.pdf
+
+SUSP 1.12
+  ftp://ftp.ymi.com/pub/rockridge/susp112.ps
+
+RRIP 1.12
+  ftp://ftp.ymi.com/pub/rockridge/rrip112.ps
+
+---------------------------------------------------------------------------
+
+This text is under
+Copyright (c) 2009 - 2010, 2020 Thomas SCHMITT <scdbackup@gmx.net>
+Copyright (c) 2020 - Valentín KIVACHUK BURDÁ <vk18496@gmail.com>
+It shall reflect the effective technical specifications as implemented in
+zisofs2-tools and the Linux kernel. So please contact mailing list
+<bug-xorriso@gnu.org> or to the copyright holders in private, if you
+want to make changes.
+Only if you cannot reach the copyright holder for at least one month it is
+permissible to modify and distribute this text under the license "GPLv3".
+

libisofs/filters/zisofs.c

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2009 - 2011 Thomas Schmitt
+ * Copyright (c) 2009 - 2020 Thomas Schmitt
  * 
  * This file is part of the libisofs project; you can redistribute it and/or 
  * modify it under the terms of the GNU General Public License version 2 
@@ -23,6 +23,7 @@
 #include "../fsource.h"
 #include "../util.h"
 #include "../stream.h"
+#include "../messages.h"
 
 #include <sys/types.h>
 #include <sys/time.h>
@@ -44,6 +45,45 @@
  */
 
 
+/* The lowest size of a file which shall not be represented by zisofs v1 */
+#define ISO_ZISOFS_V1_LIMIT 4294967296
+
+/* zisofs2: Test value for small mixed-version ISOs: 1 million
+            ISO_ZISOFS_V1_LIMIT 1000000
+*/
+
+/* Limit for overall count of allocated block pointers:
+   2 exp 25 = 256 MiB blocklist buffer = 4 TiB uncompressed at 128 KiB
+*/
+#define ISO_ZISOFS_MAX_BLOCKS_T 0x2000000
+
+/* Function to account for global block pointers */
+static uint64_t ziso_block_pointer_mgt(uint64_t num, int mode);
+
+/* Limit for single files:
+   2 exp 25 = 256 MiB blocklist buffer = 4 TiB uncompressed at 128 KiB
+*/
+#define ISO_ZISOFS_MAX_BLOCKS_F 0x2000000
+
+#ifdef Not_yeT
+
+/* The number of blocks from which on the block pointer list shall be discarded
+ * on iso_stream_close() of a compressing stream. This means that list and size
+ * have to be determined again on next ziso_stream_get_size().
+ * zisofs v1 with uint32_t pointers could at most have 131072 pointers.
+ * Since pointers are now uint64_t, the limit tolerates half of this.
+ */ 
+#define ISO_ZISOFS_MANY_BLOCKS 65537
+
+#endif /* Not_yeT */
+
+/* Minimum and maximum blocks sizes for version 1 and 2 */
+#define ISO_ZISOFS_V1_MIN_LOG2   15
+#define ISO_ZISOFS_V1_MAX_LOG2   17
+#define ISO_ZISOFS_V2_MIN_LOG2   15
+#define ISO_ZISOFS_V2_MAX_LOG2   20
+
+
 /* --------------------------- ZisofsFilterRuntime ------------------------- */
 
 
@@ -51,6 +91,13 @@
 static uint8_t ziso_block_size_log2 = 15;
 static int ziso_block_size = 32768;
 
+static int ziso_v2_enabled = 0;
+static int ziso_v2_block_size_log2 = 17;
+static int ziso_v2_block_size = 1 << 17;
+
+static int64_t ziso_max_total_blocks = ISO_ZISOFS_MAX_BLOCKS_T;
+static int64_t ziso_max_file_blocks = ISO_ZISOFS_MAX_BLOCKS_F;
+
 
 /* Individual runtime properties exist only as long as the stream is opened.
  */
@@ -58,10 +105,12 @@ typedef struct
 {
     int state; /* processing: 0= header, 1= block pointers, 2= data blocks */
 
+    int zisofs_version; /* 1 or 2 */
+
     int block_size;
-    int block_pointer_fill;
-    int block_pointer_rpos;
-    uint32_t *block_pointers; /* These are in use only with uncompression.
+    int64_t block_pointer_fill;
+    int64_t block_pointer_rpos;
+    uint64_t *block_pointers; /* These are in use only with uncompression.
                                  Compression streams hold the pointer in
                                  their persistent data.
                                */
@@ -87,8 +136,10 @@ int ziso_running_destroy(ZisofsFilterRuntime **running, int flag)
     ZisofsFilterRuntime *o= *running;
     if (o == NULL)
         return 0;
-    if (o->block_pointers != NULL)
+    if (o->block_pointers != NULL) {
+        ziso_block_pointer_mgt((uint64_t) o->block_pointer_fill, 2);
         free(o->block_pointers);
+    }
     if (o->read_buffer != NULL)
         free(o->read_buffer);
     if (o->block_buffer != NULL)
@@ -101,6 +152,7 @@ int ziso_running_destroy(ZisofsFilterRuntime **running, int flag)
 
 /*
  * @param flag bit0= do not set block_size, do not allocate buffers
+ *             bit1= use ziso_v2_block_size
  */
 static
 int ziso_running_new(ZisofsFilterRuntime **running, int flag)
@@ -112,6 +164,7 @@ int ziso_running_new(ZisofsFilterRuntime **running, int flag)
     }
     o->state = 0;
     o->block_size= 0;
+    o->zisofs_version = 0;
     o->block_pointer_fill = 0;
     o->block_pointer_rpos = 0;
     o->block_pointers = NULL;
@@ -128,11 +181,14 @@ int ziso_running_new(ZisofsFilterRuntime **running, int flag)
     if (flag & 1)
         return 1;
 
-    o->block_size = ziso_block_size;
+    if (flag & 2)
+        o->block_size = ziso_v2_block_size;
+    else
+        o->block_size = ziso_block_size;
 #ifdef Libisofs_with_zliB
-    o->buffer_size = compressBound((uLong) ziso_block_size);
+    o->buffer_size = compressBound((uLong) o->block_size);
 #else
-    o->buffer_size = 2 * ziso_block_size;
+    o->buffer_size = 2 * o->block_size;
 #endif
     o->read_buffer = calloc(o->block_size, 1);
     o->block_buffer = calloc(o->buffer_size, 1);
@@ -145,12 +201,52 @@ failed:
 }
 
 
+/* --------------------------- Resource accounting ------------------------- */
+
+/* @param mode  0= inquire whether num block pointers would fit
+                1= register num block pointers
+                2= unregister num block_pointers
+                3= return number of accounted block pointers
+   @return      if not mode 3: 0= does not fit , 1= fits
+*/
+static
+uint64_t ziso_block_pointer_mgt(uint64_t num, int mode)
+{
+    static uint64_t global_count = 0;
+    static int underrun = 0;
+
+    if (mode == 2) {
+        if (global_count < num) {
+            if (underrun < 3)
+                iso_msg_submit(-1, ISO_ZISOFS_BPT_UNDERRUN, 0, 
+                            "Prevented global block pointer counter underrun");
+            underrun++;
+            global_count = 0;
+        } else {
+            global_count -= num;
+        }
+    } else if (mode == 3) {
+        return global_count;
+    } else {
+       if (global_count + num > (uint64_t) ziso_max_total_blocks)
+           return 0;
+       if (mode == 1)
+           global_count += num;
+    }
+    return 1;
+}
+
+
 /* ---------------------------- ZisofsFilterStreamData --------------------- */
 
 /* The first 8 bytes of a zisofs compressed data file */
 static unsigned char zisofs_magic[9] =
                               {0x37, 0xE4, 0x53, 0x96, 0xC9, 0xDB, 0xD6, 0x07};
 
+/* The first 8 bytes of a zisofs2 compressed data file */
+static unsigned char zisofs2_magic[9] =
+                              {0xEF, 0x22, 0x55, 0xA1, 0xBC, 0x1B, 0x95, 0xA0};
+
 /* Counts the number of active compression filters */
 static off_t ziso_ref_count = 0;
 
@@ -191,11 +287,13 @@ typedef struct
 {   
     ZisofsFilterStreamData std;
 
-    uint32_t orig_size;
-    uint32_t *block_pointers; /* Cache for output block addresses. They get
+    uint64_t orig_size;
+    uint64_t *block_pointers; /* Cache for output block addresses. They get
                                  written before the data and so need 2 passes.
                                  This cache avoids surplus passes.
                                */
+    uint64_t block_pointer_counter;
+    uint64_t open_counter;
 } ZisofsComprStreamData;
 
 
@@ -207,6 +305,7 @@ typedef struct
 {
     ZisofsFilterStreamData std;
 
+    uint8_t zisofs_algo_num;
     unsigned char header_size_div4;
     unsigned char block_size_log2;
 
@@ -227,18 +326,61 @@ static ino_t ziso_ino_id = 0;
 static
 int ziso_stream_uncompress(IsoStream *stream, void *buf, size_t desired);
 
+static
+int ziso_stream_compress(IsoStream *stream, void *buf, size_t desired);
+
+
+static
+int ziso_decide_v2_usage(off_t orig_size)
+{
+    if (ziso_v2_enabled > 1 ||
+        (ziso_v2_enabled == 1 && orig_size >= (off_t) ISO_ZISOFS_V1_LIMIT))
+        return 1;
+    return 0;
+}
+
+
 /*
  * @param flag  bit0= original stream is not open
+ *              bit1= do not destroy large
+ *                    ZisofsComprStreamData->block_pointers
  */
 static
 int ziso_stream_close_flag(IsoStream *stream, int flag)
 {
     ZisofsFilterStreamData *data;
+    ZisofsComprStreamData *cstd = NULL;
 
     if (stream == NULL) {
         return ISO_NULL_POINTER;
     }
     data = stream->data;
+    if (stream->class->read == &ziso_stream_compress)
+        cstd = (ZisofsComprStreamData *) data;
+
+#ifdef Not_yeT
+
+    /* >>> zisofs2:
+           research whether zisofs streams get opened and closed often */
+
+    if (cstd != NULL) {
+        int block_size;
+
+        block_size = ziso_block_size;
+        if (ziso_decide_v2_usage(cstd->orig_size))
+            block_size = ziso_v2_block_size;
+        if ((!(flag & 2)) && cstd->open_counter == 1 &&
+            cstd->orig_size / block_size >= ISO_ZISOFS_MANY_BLOCKS) {
+            if (cstd->block_pointers != NULL) {
+                ziso_block_pointer_mgt(cstd->block_pointer_counter, 2);
+                free((char *) cstd->block_pointers);
+            }
+            cstd->block_pointers = NULL;
+            data->size = -1;
+        }
+    }
+
+#endif /* Not_yeT */
 
     if (data->running == NULL) {
         return 1;
@@ -246,6 +388,9 @@ int ziso_stream_close_flag(IsoStream *stream, int flag)
     ziso_running_destroy(&(data->running), 0);
     if (flag & 1)
         return 1;
+    if (cstd != NULL)
+        if (cstd->open_counter > 0)
+            cstd->open_counter--;
     return iso_stream_close(data->orig);
 }
 
@@ -264,8 +409,9 @@ static
 int ziso_stream_open_flag(IsoStream *stream, int flag)
 {
     ZisofsFilterStreamData *data;
+    ZisofsComprStreamData *cstd;
     ZisofsFilterRuntime *running = NULL;
-    int ret;
+    int ret, use_v2 = 0;
 
     if (stream == NULL) {
         return ISO_NULL_POINTER;
@@ -280,9 +426,15 @@ int ziso_stream_open_flag(IsoStream *stream, int flag)
         */
         stream->class->get_size(stream);
     }
+    if (stream->class->read == &ziso_stream_compress) {
+        cstd = (ZisofsComprStreamData *) data;
+        cstd->open_counter++;
+        use_v2 = ziso_decide_v2_usage((off_t) cstd->orig_size);
+    }
 
     ret = ziso_running_new(&running,
-                           stream->class->read == &ziso_stream_uncompress);
+                           (stream->class->read == &ziso_stream_uncompress) |
+                           ((!!use_v2) << 1));
     if (ret < 0) {
         return ret;
     }
@@ -316,6 +468,7 @@ int ziso_stream_compress(IsoStream *stream, void *buf, size_t desired)
     off_t orig_size, next_pt;
     char *cbuf = buf;
     uLongf buf_len;
+    uint64_t *copy_base, num_blocks = 0;
 
     if (stream == NULL) {
         return ISO_NULL_POINTER;
@@ -334,18 +487,39 @@ int ziso_stream_compress(IsoStream *stream, void *buf, size_t desired)
             /* Delivering file header */
 
             if (rng->buffer_fill == 0) {
-                memcpy(rng->block_buffer, zisofs_magic, 8);
                 orig_size = iso_stream_get_size(data->std.orig);
-                if (orig_size > 4294967295.0) {
+                num_blocks = orig_size / rng->block_size +
+                             1 + !!(orig_size % rng->block_size);
+                if (num_blocks > (uint64_t) ziso_max_file_blocks)
                     return (rng->error_ret = ISO_ZISOFS_TOO_LARGE);
+                if (ziso_block_pointer_mgt((uint64_t) num_blocks, 0) == 0)
+                   return (rng->error_ret = ISO_ZISOFS_TOO_MANY_PTR);
+                if (orig_size != (off_t) data->orig_size)
+                    return (rng->error_ret = ISO_FILTER_WRONG_INPUT);
+                if (ziso_decide_v2_usage(orig_size)) {
+                    rng->zisofs_version = 2;
+                    memcpy(rng->block_buffer, zisofs2_magic, 8);
+                    rng->block_buffer[8] = 0;    /* @hdr_version */
+                    rng->block_buffer[9] = 6;    /* @hdr_size */
+                    rng->block_buffer[10] = 1;   /* @alg_id */
+                    rng->block_buffer[11] = ziso_v2_block_size_log2;
+                    iso_lsb64((uint8_t *) (rng->block_buffer + 12),
+                              (uint64_t) orig_size);
+                    memset(rng->block_buffer + 20, 0, 4);
+                    rng->buffer_fill = 24;
+                } else {
+                    if (orig_size >= (off_t) ISO_ZISOFS_V1_LIMIT) {
+                        return (rng->error_ret = ISO_ZISOFS_TOO_LARGE);
+                    }
+                    rng->zisofs_version = 1;
+                    memcpy(rng->block_buffer, zisofs_magic, 8);
+                    iso_lsb((unsigned char *) (rng->block_buffer + 8),
+                            (uint32_t) orig_size, 4);
+                    rng->block_buffer[12] = 4;
+                    rng->block_buffer[13] = ziso_block_size_log2;
+                    rng->block_buffer[14] = rng->block_buffer[15] = 0;
+                    rng->buffer_fill = 16;
                 }
-                data->orig_size = orig_size;
-                iso_lsb((unsigned char *) (rng->block_buffer + 8),
-                        (uint32_t) orig_size, 4);
-                rng->block_buffer[12] = 4;
-                rng->block_buffer[13] = ziso_block_size_log2;
-                rng->block_buffer[14] = rng->block_buffer[15] = 0;
-                rng->buffer_fill = 16;
                 rng->buffer_rpos = 0;
             } else if (rng->buffer_rpos >= rng->buffer_fill) {
                 rng->buffer_fill = rng->buffer_rpos = 0;
@@ -358,17 +532,25 @@ int ziso_stream_compress(IsoStream *stream, void *buf, size_t desired)
             if (rng->block_pointer_fill == 0) {
                 /* Initialize block pointer writing */
                 rng->block_pointer_rpos = 0;
-                rng->block_pointer_fill = data->orig_size / rng->block_size
-                                   + 1 + !!(data->orig_size % rng->block_size);
+                num_blocks = data->orig_size / rng->block_size
+                             + 1 + !!(data->orig_size % rng->block_size);
+                rng->block_pointer_fill = num_blocks;
                 if (data->block_pointers == NULL) {
                     /* On the first pass, create pointer array with all 0s */
-                    data->block_pointers = calloc(rng->block_pointer_fill, 4);
+                    if (ziso_block_pointer_mgt(num_blocks, 1) == 0) {
+                        rng->block_pointer_fill = 0;
+                        return (rng->error_ret = ISO_ZISOFS_TOO_MANY_PTR);
+                    }
+                    data->block_pointers = calloc(rng->block_pointer_fill, 8);
                     if (data->block_pointers == NULL) {
+                        ziso_block_pointer_mgt(num_blocks, 2);
                         rng->block_pointer_fill = 0;
                         return (rng->error_ret = ISO_OUT_OF_MEM);
                     }
+                    data->block_pointer_counter = rng->block_pointer_fill;
                 }
             }
+
             if (rng->buffer_rpos >= rng->buffer_fill) {
                 if (rng->block_pointer_rpos >= rng->block_pointer_fill) {
                     rng->buffer_fill = rng->buffer_rpos = 0;
@@ -377,14 +559,26 @@ int ziso_stream_compress(IsoStream *stream, void *buf, size_t desired)
                     rng->state = 2; /* block pointers are delivered */
                 } else {
                     /* Provide a buffer full of block pointers */
+                    /* data->block_pointers was filled by ziso_stream_open() */
                     todo = rng->block_pointer_fill - rng->block_pointer_rpos;
-                    if (todo * 4 > rng->buffer_size)
-                        todo = rng->buffer_size / 4;
-                    memcpy(rng->block_buffer,
-                           data->block_pointers + rng->block_pointer_rpos,
-                           todo * 4);
+                    copy_base = data->block_pointers + rng->block_pointer_rpos;
+                    if (rng->zisofs_version == 1) {
+                        if (todo * 4 > rng->buffer_size)
+                            todo = rng->buffer_size / 4;
+                        for (i = 0; i < todo; i++)
+                            iso_lsb((unsigned char *) (rng->block_buffer +
+                                                       4 * i), 
+                                    (uint32_t) (copy_base[i] & 0xffffffff), 4);
+                        rng->buffer_fill = todo * 4;
+                    } else {
+                        if (todo * 8 > rng->buffer_size)
+                            todo = rng->buffer_size / 8;
+                        for (i = 0; i < todo; i++)
+                            iso_lsb64((uint8_t *) rng->block_buffer + 8 * i,
+                                      copy_base[i]);
+                        rng->buffer_fill = todo * 8;
+                    }
                     rng->buffer_rpos = 0;
-                    rng->buffer_fill = todo * 4;
                     rng->block_pointer_rpos += todo;
                 }
             }
@@ -396,7 +590,7 @@ int ziso_stream_compress(IsoStream *stream, void *buf, size_t desired)
                                   rng->block_size);
             if (ret > 0) {
                 rng->in_counter += ret;
-                if (rng->in_counter > data->orig_size) {
+                if ((uint64_t) rng->in_counter > data->orig_size) {
                     /* Input size became larger */
                     return (rng->error_ret = ISO_FILTER_WRONG_INPUT);
                 }
@@ -425,10 +619,11 @@ int ziso_stream_compress(IsoStream *stream, void *buf, size_t desired)
                     return (rng->error_ret = ISO_FILTER_WRONG_INPUT);
                 }
 
-                /* Record resp. check block pointer */
+                /* Check or record check block pointer */
                 rng->block_counter++;
                 if (data->block_pointers[rng->block_counter] > 0) {
-                    if (next_pt != data->block_pointers[rng->block_counter] ) {
+                    if ((uint64_t) next_pt !=
+                        data->block_pointers[rng->block_counter]) {
                         /* block pointers mismatch , content has changed */
                         return (rng->error_ret = ISO_FILTER_WRONG_INPUT);
                     }
@@ -438,7 +633,7 @@ int ziso_stream_compress(IsoStream *stream, void *buf, size_t desired)
 
             } else if (ret == 0) {
                 rng->state = 3;
-                if (rng->in_counter != data->orig_size) {
+                if ((uint64_t) rng->in_counter != data->orig_size) {
                     /* Input size shrunk */
                     return (rng->error_ret = ISO_FILTER_WRONG_INPUT);
                 }
@@ -478,24 +673,91 @@ int ziso_stream_compress(IsoStream *stream, void *buf, size_t desired)
 
 
 static
-int ziso_parse_zisofs_head(IsoStream *stream, int *header_size_div4,
-                           int *block_size_log2, uint32_t *uncompressed_size,
-                           int flag)
+int ziso_algo_to_num(uint8_t zisofs_algo[2])
 {
-    int ret;
-    char zisofs_head[16];
+    if (zisofs_algo[0] == 'p' && zisofs_algo[1] == 'z')
+        return 0;
+    if (zisofs_algo[0] == 'P' && zisofs_algo[1] == 'Z')
+        return 1;
+    return -1;
+}
 
-    ret = iso_stream_read(stream, zisofs_head, 16);
+
+static
+int ziso_num_to_algo(uint8_t num, uint8_t zisofs_algo[2])
+{
+    if (num == 0) {
+        zisofs_algo[0] = 'p';
+        zisofs_algo[1] = 'z';
+        return 1;
+    } else if (num == 1) {
+        zisofs_algo[0] = 'P';
+        zisofs_algo[1] = 'Z';
+        return 2;
+    }
+    return -1;
+}
+
+
+/* @param flag bit0= recognize zisofs2 only if ziso_v2_enabled
+*/
+static
+int ziso_parse_zisofs_head(IsoStream *stream, uint8_t *ziso_algo_num,
+                           int *header_size_div4, int *block_size_log2,
+                           uint64_t *uncompressed_size, int flag)
+{
+    int ret, consumed = 0, i;
+    char zisofs_head[24];
+    char waste_word[4];
+
+    ret = iso_stream_read(stream, zisofs_head, 8);
     if (ret < 0)
         return ret;
-    *header_size_div4 = ((unsigned char *) zisofs_head)[12];
-    *block_size_log2 = ((unsigned char *) zisofs_head)[13];
-    if (ret != 16 || memcmp(zisofs_head, zisofs_magic, 8) != 0 ||
-        *header_size_div4 < 4 ||
-        *block_size_log2 < 15 || *block_size_log2 > 17) {
+    if (ret != 8)
+        return ISO_ZISOFS_WRONG_INPUT;
+    consumed = 8;
+    if (memcmp(zisofs_head, zisofs_magic, 8) == 0) {
+        *ziso_algo_num = 0;
+        ret = iso_stream_read(stream, zisofs_head + 8, 8);
+        if (ret < 0)
+            return ret;
+        if (ret != 8)
+            return ISO_ZISOFS_WRONG_INPUT;
+        consumed += 8;
+        *header_size_div4 = ((unsigned char *) zisofs_head)[12];
+        *block_size_log2 = ((unsigned char *) zisofs_head)[13];
+        *uncompressed_size = iso_read_lsb(((uint8_t *) zisofs_head) + 8, 4);
+        if (*header_size_div4 < 4 ||
+            *block_size_log2 < ISO_ZISOFS_V1_MIN_LOG2 ||
+            *block_size_log2 > ISO_ZISOFS_V1_MAX_LOG2)
+            return ISO_ZISOFS_WRONG_INPUT;
+    } else if (memcmp(zisofs_head, zisofs2_magic, 8) == 0 &&
+               !(ziso_v2_enabled == 0 && (flag & 1))) {
+        ret = iso_stream_read(stream, zisofs_head + 8, 16);
+        if (ret < 0)
+            return ret;
+        if (ret != 16)
+            return ISO_ZISOFS_WRONG_INPUT;
+        consumed += 16;
+        *ziso_algo_num = zisofs_head[10];
+        *header_size_div4 = ((unsigned char *) zisofs_head)[9];
+        *block_size_log2 = ((unsigned char *) zisofs_head)[11];
+        *uncompressed_size = iso_read_lsb64(((uint8_t *) zisofs_head) + 12);
+        if (*header_size_div4 < 4 ||
+            *block_size_log2 < ISO_ZISOFS_V2_MIN_LOG2 ||
+            *block_size_log2 > ISO_ZISOFS_V2_MAX_LOG2 || *ziso_algo_num != 1)
+            return ISO_ZISOFS_WRONG_INPUT;
+    } else {
         return ISO_ZISOFS_WRONG_INPUT;
     }
-    *uncompressed_size = iso_read_lsb(((uint8_t *) zisofs_head) + 8, 4);
+    for (i = consumed; i < *header_size_div4; i++) {
+       /* Skip surplus header words */
+       ret = iso_stream_read(stream, waste_word, 4);
+       if (ret < 0)
+           return ret;
+       if (ret != 4)
+           return ISO_ZISOFS_WRONG_INPUT; 
+    }
     return 1;
 }
 
@@ -509,15 +771,16 @@ int ziso_stream_uncompress(IsoStream *stream, void *buf, size_t desired)
 
 #ifdef Libisofs_with_zliB
 
-    int ret, todo, i, header_size, bs_log2, block_max = 1;
+    int ret, todo, header_size, bs_log2, block_max = 1, blpt_size;
     ZisofsFilterStreamData *data;
     ZisofsFilterRuntime *rng;
     ZisofsUncomprStreamData *nstd;
     size_t fill = 0;
     char *cbuf = buf;
     uLongf buf_len;
-    uint32_t uncompressed_size;
-    char waste_word[4];
+    uint64_t uncompressed_size;
+    int64_t i;
+    uint8_t algo_num, *rpt, *wpt;
 
     if (stream == NULL) {
         return ISO_NULL_POINTER;
@@ -535,46 +798,63 @@ int ziso_stream_uncompress(IsoStream *stream, void *buf, size_t desired)
     while (1) {
         if (rng->state == 0) {
             /* Reading file header */
-            ret = ziso_parse_zisofs_head(data->orig, &header_size, &bs_log2,
-                                         &uncompressed_size, 0);
+            ret = ziso_parse_zisofs_head(data->orig, &algo_num, &header_size,
+                                         &bs_log2, &uncompressed_size, 0);
             if (ret < 0)
                 return (rng->error_ret = ret);
+            if (algo_num == 0)
+                blpt_size = 4;
+            else
+                blpt_size = 8;
             nstd->header_size_div4 = header_size;
             header_size *= 4;
             data->size = uncompressed_size;
             nstd->block_size_log2 = bs_log2;
             rng->block_size = 1 << bs_log2;
-            for (i = 16; i < header_size; i += 4) {
-                 /* Skip surplus header words */
-                 ret = iso_stream_read(data->orig, waste_word, 4);
-                 if (ret < 0)
-                     return (rng->error_ret = ret);
-                 if (ret != 4)
-                     return (rng->error_ret = ISO_ZISOFS_WRONG_INPUT); 
-            }
 
-            if (desired == 0) {
+            if (desired == 0)
                 return 0;
-            }
 
             /* Create and read pointer array */
             rng->block_pointer_rpos = 0;
             rng->block_pointer_fill = data->size / rng->block_size
                                      + 1 + !!(data->size % rng->block_size);
-            rng->block_pointers = calloc(rng->block_pointer_fill, 4);
+            if (rng->block_pointer_fill > ziso_max_file_blocks) {
+                rng->block_pointer_fill = 0;
+                return (rng->error_ret = ISO_ZISOFS_TOO_LARGE);
+            }
+            if (ziso_block_pointer_mgt((uint64_t) rng->block_pointer_fill, 1)
+                == 0)
+                return ISO_ZISOFS_TOO_MANY_PTR;
+            rng->block_pointers = calloc(rng->block_pointer_fill, 8);
             if (rng->block_pointers == NULL) {
+                ziso_block_pointer_mgt((uint64_t) rng->block_pointer_fill, 2);
                 rng->block_pointer_fill = 0;
                 return (rng->error_ret = ISO_OUT_OF_MEM);
             }
             ret = iso_stream_read(data->orig, rng->block_pointers,
-                                  rng->block_pointer_fill * 4);
+                                  rng->block_pointer_fill * blpt_size);
             if (ret < 0)
                 return (rng->error_ret = ret);
-            if (ret != rng->block_pointer_fill * 4)
+            if (algo_num == 0) {
+                /* Spread 4 byte little-endian pointer values over 8 byte */
+                rpt = ((uint8_t *) rng->block_pointers)
+                      + rng->block_pointer_fill * 4;
+                wpt = ((uint8_t *) rng->block_pointers)
+                      + rng->block_pointer_fill * 8;
+		while (rpt > ((uint8_t *) rng->block_pointers) + 4) {
+                    rpt -= 4;
+                    wpt -= 8;
+                    memcpy(wpt, rpt, 4);
+                    memset(wpt + 4, 0, 4);
+                }
+                memset(((uint8_t *) rng->block_pointers) + 4, 0, 4);
+            }
+            if (ret != rng->block_pointer_fill * blpt_size)
                return (rng->error_ret = ISO_ZISOFS_WRONG_INPUT);
             for (i = 0; i < rng->block_pointer_fill; i++) {
                  rng->block_pointers[i] =
-                      iso_read_lsb((uint8_t *) (rng->block_pointers + i), 4);
+                         iso_read_lsb64((uint8_t *) (rng->block_pointers + i));
                  if (i > 0)
                      if ((int) (rng->block_pointers[i] -
                                 rng->block_pointers[i - 1])
@@ -704,7 +984,7 @@ off_t ziso_stream_get_size(IsoStream *stream)
             count += ret;
         }
     }
-    ret_close = ziso_stream_close(stream);
+    ret_close = ziso_stream_close_flag(stream, 2);
     if (ret < 0)
         return ret;
     if (ret_close < 0)
@@ -754,8 +1034,10 @@ void ziso_stream_free(IsoStream *stream)
             ziso_osiz_ref_count = 0;
     } else {
         nstd = stream->data;
-        if (nstd->block_pointers != NULL)
+        if (nstd->block_pointers != NULL) {
+            ziso_block_pointer_mgt(nstd->block_pointer_counter, 2);
             free((char *) nstd->block_pointers);
+        }
         if (--ziso_ref_count < 0)
             ziso_ref_count = 0;
     }
@@ -807,6 +1089,7 @@ int ziso_clone_stream(IsoStream *old_stream, IsoStream **new_stream, int flag)
             goto no_mem;
         stream_data = (ZisofsFilterStreamData *) uncompr;
         old_uncompr = (ZisofsUncomprStreamData *) old_stream->data;
+        uncompr->zisofs_algo_num = old_uncompr->zisofs_algo_num;
         uncompr->header_size_div4 = old_uncompr->header_size_div4;
         uncompr->block_size_log2 = old_uncompr->block_size_log2;
     } else {
@@ -817,6 +1100,8 @@ int ziso_clone_stream(IsoStream *old_stream, IsoStream **new_stream, int flag)
         old_compr = (ZisofsComprStreamData *) old_stream->data;
         compr->orig_size = old_compr->orig_size;
         compr->block_pointers = NULL;
+        compr->block_pointer_counter = 0;
+        compr->open_counter = 0;
     }
     old_stream_data = (ZisofsFilterStreamData *) old_stream->data;
     stream_data->orig = new_input_stream;
@@ -971,13 +1256,16 @@ int ziso_filter_get_filter(FilterContext *filter, IsoStream *original,
     str->refcount = 1;
     str->data = data;
     if (flag & 2) {
+        unstd->zisofs_algo_num = 0;
         unstd->header_size_div4 = 0;
         unstd->block_size_log2 = 0;
         str->class = &ziso_stream_uncompress_class;
         ziso_osiz_ref_count++;
     } else {
-        cnstd->orig_size = 0;
+        cnstd->orig_size = iso_stream_get_size(original);
         cnstd->block_pointers = NULL;
+        cnstd->block_pointer_counter = 0;
+        cnstd->open_counter = 0;
         str->class = &ziso_stream_compress_class;
         ziso_ref_count++;
     }
@@ -1059,7 +1347,7 @@ int ziso_add_filter(IsoFile *file, int flag)
         if (original_size <= 0 || ((flag & 1) && original_size <= 2048)) {
             return 2;
         }
-        if (original_size > 4294967295.0) {
+        if (original_size >= (off_t) ISO_ZISOFS_V1_LIMIT && !ziso_v2_enabled) {
             return ISO_ZISOFS_TOO_LARGE;
         }
     }
@@ -1119,9 +1407,9 @@ int iso_zisofs_get_refcounts(off_t *ziso_count, off_t *osiz_count, int flag)
 }
 
 
-int ziso_add_osiz_filter(IsoFile *file, uint8_t header_size_div4,
-                         uint8_t block_size_log2, uint32_t uncompressed_size,
-                         int flag)
+int ziso_add_osiz_filter(IsoFile *file, uint8_t zisofs_algo[2],
+                         uint8_t header_size_div4, uint8_t block_size_log2,
+                         uint64_t uncompressed_size, int flag)
 {   
 
 #ifdef Libisofs_with_zliB
@@ -1133,6 +1421,10 @@ int ziso_add_osiz_filter(IsoFile *file, uint8_t header_size_div4,
     if (ret < 0)
         return ret;
     unstd = iso_file_get_stream(file)->data;
+    ret = ziso_algo_to_num(zisofs_algo);
+    if (ret < 0)
+        return ISO_ZISOFS_WRONG_INPUT;
+    unstd->zisofs_algo_num = ret;
     unstd->header_size_div4 = header_size_div4;
     unstd->block_size_log2 = block_size_log2;
     unstd->std.size = uncompressed_size;
@@ -1152,28 +1444,44 @@ int ziso_add_osiz_filter(IsoFile *file, uint8_t header_size_div4,
    and eventual ZF field parameters
    @param flag bit0= allow ziso_by_content which is based on content reading
                bit1= do not inquire stream->class for filters
+               bit2= recognize zisofs2 by magic only if ziso_v2_enabled
 */
 int ziso_is_zisofs_stream(IsoStream *stream, int *stream_type,
+                          uint8_t zisofs_algo[2],
                           int *header_size_div4, int *block_size_log2,
-                          uint32_t *uncompressed_size, int flag)
+                          uint64_t *uncompressed_size, int flag)
 {
-    int ret, close_ret;
+    int ret, close_ret, algo_ret;
     ZisofsFilterStreamData *data;
     ZisofsComprStreamData *cnstd;
     ZisofsUncomprStreamData *unstd;
+    uint8_t algo_num;
 
     *stream_type = 0; 
     if (stream->class == &ziso_stream_compress_class && !(flag & 2)) {
         *stream_type = 1;
         cnstd = stream->data;
         *header_size_div4 = 4;
-        *block_size_log2 = ziso_block_size_log2;
         *uncompressed_size = cnstd->orig_size;
+        if (ziso_decide_v2_usage((off_t) *uncompressed_size)) {
+          *block_size_log2 = ziso_v2_block_size_log2;
+          zisofs_algo[0] = 'P';
+          zisofs_algo[1] = 'Z';
+        } else if (*uncompressed_size < (uint64_t) ISO_ZISOFS_V1_LIMIT) {
+          *block_size_log2 = ziso_block_size_log2;
+          zisofs_algo[0] = 'p';
+          zisofs_algo[1] = 'z';
+        } else {
+          return 0;
+        }
         return 1;
     } else if(stream->class == &ziso_stream_uncompress_class && !(flag & 2)) {
         *stream_type = -1;
         data = stream->data;
         unstd = stream->data;
+        ret = ziso_num_to_algo(unstd->zisofs_algo_num, zisofs_algo);
+        if (ret < 0) 
+            return ISO_ZISOFS_WRONG_INPUT;
         *header_size_div4 = unstd->header_size_div4;
         *block_size_log2 = unstd->block_size_log2;
         *uncompressed_size = data->size;
@@ -1185,14 +1493,19 @@ int ziso_is_zisofs_stream(IsoStream *stream, int *stream_type,
     ret = iso_stream_open(stream);
     if (ret < 0) 
         return ret;
-    ret = ziso_parse_zisofs_head(stream, header_size_div4,
-                                 block_size_log2, uncompressed_size, 0);
+    ret = ziso_parse_zisofs_head(stream, &algo_num, header_size_div4,
+                                 block_size_log2, uncompressed_size,
+                                 (flag >> 2) & 1);
     if (ret == 1) {
         *stream_type = 2;
+        algo_ret = ziso_num_to_algo(algo_num, zisofs_algo);
     } else {
         ret = 0;
+        algo_ret = 1;
     }
     close_ret = iso_stream_close(stream);
+    if (algo_ret < 0) 
+        return ISO_ZISOFS_WRONG_INPUT;
     if (close_ret < 0) 
         return close_ret;
 
@@ -1200,21 +1513,47 @@ int ziso_is_zisofs_stream(IsoStream *stream, int *stream_type,
 }
 
 
+/* API */
 int iso_zisofs_set_params(struct iso_zisofs_ctrl *params, int flag)
 {
 
 #ifdef Libisofs_with_zliB
 
+    if (params->version < 0 || params->version > 1)
+       return ISO_WRONG_ARG_VALUE;
+
     if (params->compression_level < 0 || params->compression_level > 9 ||
-        params->block_size_log2 < 15 || params->block_size_log2  > 17) {
+        params->block_size_log2 < ISO_ZISOFS_V1_MIN_LOG2 ||
+        params->block_size_log2  > ISO_ZISOFS_V1_MAX_LOG2) {
         return ISO_WRONG_ARG_VALUE;
     }
+    if (params->version >= 1)
+        if (params->v2_enabled < 0 || params->v2_enabled > 2 ||
+            (params->v2_block_size_log2 != 0 &&
+             (params->v2_block_size_log2 < ISO_ZISOFS_V2_MIN_LOG2 ||
+              params->v2_block_size_log2 > ISO_ZISOFS_V2_MAX_LOG2)))
+            return ISO_WRONG_ARG_VALUE;
     if (ziso_ref_count > 0) {
         return ISO_ZISOFS_PARAM_LOCK;
     }
     ziso_compression_level = params->compression_level;
     ziso_block_size_log2 = params->block_size_log2;
     ziso_block_size = 1 << ziso_block_size_log2;
+
+    if (params->version == 0)
+        return 1;
+
+    ziso_v2_enabled = params->v2_enabled;
+    if (params->v2_block_size_log2 > 0)
+        ziso_v2_block_size_log2 = params->v2_block_size_log2;
+    ziso_v2_block_size = 1 << ziso_v2_block_size_log2;
+    if (params->max_total_blocks > 0)
+        ziso_max_total_blocks = params->max_total_blocks;
+    if (params->max_file_blocks > 0)
+        ziso_max_file_blocks = params->max_file_blocks;
+
+    /* >>> zisofs2: more parameters */
+
     return 1;
     
 #else
@@ -1226,13 +1565,24 @@ int iso_zisofs_set_params(struct iso_zisofs_ctrl *params, int flag)
 }
 
 
+/* API */
 int iso_zisofs_get_params(struct iso_zisofs_ctrl *params, int flag)
 {
 
 #ifdef Libisofs_with_zliB
 
+    if (params->version < 0 || params->version > 1)
+       return ISO_WRONG_ARG_VALUE;
+
     params->compression_level = ziso_compression_level;
     params->block_size_log2 = ziso_block_size_log2;
+    if (params->version == 1) {
+        params->v2_enabled = ziso_v2_enabled;
+        params->v2_block_size_log2 = ziso_v2_block_size_log2;
+        params->max_total_blocks = ziso_max_total_blocks;
+        params->current_total_blocks = ziso_block_pointer_mgt((uint64_t) 0, 3);
+        params->max_file_blocks = ziso_max_file_blocks;
+    }
     return 1;
 
 #else
@@ -1243,3 +1593,21 @@ int iso_zisofs_get_params(struct iso_zisofs_ctrl *params, int flag)
     
 }
 
+
+/* API */
+int iso_stream_get_zisofs_par(IsoStream *stream, int *stream_type,
+                              uint8_t zisofs_algo[2], uint8_t* algo_num,
+                              int *block_size_log2, int flag)
+{
+    uint64_t uncompressed_size;
+    int header_size_div4, ret;
+    
+    ret = ziso_is_zisofs_stream(stream, stream_type, zisofs_algo,
+                                &header_size_div4, block_size_log2,
+                                &uncompressed_size, 0);
+    if (ret <= 0 || (*stream_type != -1 && *stream_type != 1))
+        return 0;
+    *algo_num = ziso_algo_to_num(zisofs_algo);
+    return 1;
+}
+

libisofs/fs_image.c

@@ -418,9 +418,10 @@ struct image_fs_data
     unsigned int opened : 2; /**< 0 not opened, 1 opened file, 2 opened dir */
 
 #ifdef Libisofs_with_zliB
+    uint8_t zisofs_algo[2];
     uint8_t header_size_div4;
     uint8_t block_size_log2;
-    uint32_t uncompressed_size;
+    uint64_t uncompressed_size;
 #endif
 
     /* info for content reading */
@@ -1252,9 +1253,9 @@ IsoFileSourceIface ifs_class = {
 
 
 /* Used from libisofs/stream.c : iso_stream_get_src_zf() */
-int iso_ifs_source_get_zf(IsoFileSource *src, int *header_size_div4,
-                          int *block_size_log2, uint32_t *uncompressed_size,
-                          int flag)
+int iso_ifs_source_get_zf(IsoFileSource *src, uint8_t zisofs_algo[2],
+                          int *header_size_div4, int *block_size_log2, 
+                          uint64_t *uncompressed_size, int flag)
 {
 
 #ifdef Libisofs_with_zliB
@@ -1264,6 +1265,8 @@ int iso_ifs_source_get_zf(IsoFileSource *src, int *header_size_div4,
     if (src->class != &ifs_class)
         return 0;
     data = src->data;
+    zisofs_algo[0] = data->zisofs_algo[0];
+    zisofs_algo[1] = data->zisofs_algo[1];
     *header_size_div4 = data->header_size_div4;
     *block_size_log2 = data->block_size_log2;
     *uncompressed_size = data->uncompressed_size;
@@ -1460,7 +1463,7 @@ int iso_file_source_new_ifs(IsoImageFilesystem *fs, IsoFileSource *parent,
 
 #ifdef Libisofs_with_zliB
     uint8_t zisofs_alg[2], zisofs_hs4 = 0, zisofs_bsl2 = 0;
-    uint32_t zisofs_usize = 0;
+    uint64_t zisofs_usize = 0;
 #endif
 
     if (fs == NULL || fs->data == NULL || record == NULL || src == NULL) {
@@ -1593,7 +1596,7 @@ int iso_file_source_new_ifs(IsoImageFilesystem *fs, IsoFileSource *parent,
         while ((ret = susp_iter_next(iter, &sue, 0)) > 0) {
 
             /* ignore entries from different version */
-            if (sue->version[0] != 1)
+            if (sue->version[0] != 1 && !(SUSP_SIG(sue, 'Z', 'F')))
                 continue;
 
             if (SUSP_SIG(sue, 'P', 'X')) {
@@ -1804,13 +1807,23 @@ if (name != NULL && !namecont) {
 
                 ret = read_zisofs_ZF(sue, zisofs_alg, &zisofs_hs4,
                                      &zisofs_bsl2, &zisofs_usize, 0);
-                if (ret < 0 || zisofs_alg[0] != 'p' || zisofs_alg[1] != 'z') {
+                if (ret < 0) {
+invalid_zf:
                     /* notify and continue */
                     ret = iso_rr_msg_submit(fsdata, 13, ISO_WRONG_RR_WARN, ret,
                                  "Invalid ZF entry");
                     zisofs_hs4 = 0;
                     continue;
                 }
+                if (zisofs_alg[0] == 'p' || zisofs_alg[1] == 'z') {
+                   if (sue->version[0] != 1)
+                       goto invalid_zf;
+               } else if (zisofs_alg[0] == 'P' || zisofs_alg[1] == 'Z') {
+                   if (sue->version[0] != 2)
+                       goto invalid_zf;
+               } else {
+                   goto invalid_zf;
+               }
 
 #endif /* Libisofs_with_zliB */
 
@@ -2085,6 +2098,8 @@ if (name != NULL && !namecont) {
 
 #ifdef Libisofs_with_zliB
     if (zisofs_hs4 > 0) {
+        ifsdata->zisofs_algo[0] = zisofs_alg[0];
+        ifsdata->zisofs_algo[1] = zisofs_alg[1];
         ifsdata->header_size_div4 = zisofs_hs4;
         ifsdata->block_size_log2 = zisofs_bsl2;
         ifsdata->uncompressed_size = zisofs_usize;
@@ -3172,8 +3187,8 @@ int image_builder_create_node(IsoNodeBuilder *builder, IsoImage *image,
 
 #ifdef Libisofs_with_zliB
     /* Intimate friendship with this function in filters/zisofs.c */
-    int ziso_add_osiz_filter(IsoFile *file, uint8_t header_size_div4,
-                             uint8_t block_size_log2,
+    int ziso_add_osiz_filter(IsoFile *file, uint8_t zisofs_algo[2],
+                             uint8_t header_size_div4, uint8_t block_size_log2,
                              uint32_t uncompressed_size, int flag);
 #endif /* Libisofs_with_zliB */
 
@@ -3288,7 +3303,8 @@ int image_builder_create_node(IsoNodeBuilder *builder, IsoImage *image,
 #ifdef Libisofs_with_zliB
 
                 if (data->header_size_div4 > 0) {
-                    ret = ziso_add_osiz_filter(file, data->header_size_div4,
+                    ret = ziso_add_osiz_filter(file, data->zisofs_algo,
+                                               data->header_size_div4,
                                                data->block_size_log2,
                                                data->uncompressed_size, 0);
                     if (ret < 0) {

libisofs/libisofs.h

@@ -4,7 +4,7 @@
 
 /*
  * Copyright (c) 2007-2008 Vreixo Formoso, Mario Danic
- * Copyright (c) 2009-2019 Thomas Schmitt
+ * Copyright (c) 2009-2020 Thomas Schmitt
  *
  * This file is part of the libisofs project; you can redistribute it and/or
  * modify it under the terms of the GNU General Public License version 2 
@@ -7942,7 +7942,7 @@ int iso_stream_get_external_filter(IsoStream *stream,
 /**
  * Install a zisofs filter on top of the content stream of a data file.
  * zisofs is a compression format which is decompressed by some Linux kernels.
- * See also doc/zisofs_format.txt .
+ * See also doc/zisofs_format.txt and doc/zisofs2_format.txt.
  * The filter will not be installed if its output size is not smaller than
  * the size of the input stream.
  * This is only enabled if the use of libz was enabled at compile time.
@@ -7964,6 +7964,37 @@ int iso_stream_get_external_filter(IsoStream *stream,
  */
 int iso_file_add_zisofs_filter(IsoFile *file, int flag);
 
+
+/**
+ * Obtain the parameters of a zisofs filter stream.
+ * @param stream
+ *      The stream to be inquired.
+ * @param stream_type
+ *      1=compressing ("ziso")
+ *     -1=uncompressing ("osiz")
+ *      0 other (any obtained parameters have invalid content)
+ * @param zisofs_algo
+ *      Algorithm as of ZF field:
+ *      {'p', 'z'} = zisofs version 1 (Zlib)
+ *      {'P', 'Z'} = zisofs version 2 (Zlib)
+ * @param algo_num
+ *      Algorithm as of zisofs header:
+ *      0 = zisofs version 1 (Zlib)
+ *      1 = zisofs version 2 (Zlib)
+ * @param block_size_log2
+ *      Log2 of the compression block size
+ *      15 = 32 kiB , 16 = 64 kiB , 17 = 128 kiB, ...
+ * @param flag
+ *      Bitfield for control purposes, unused yet, submit 0
+ * @return
+ *      1 on success, 0 if the stream has not class->type "ziso" or "osiz"
+ * @since 1.5.4
+ */
+int iso_stream_get_zisofs_par(IsoStream *stream, int *stream_type,
+                              uint8_t zisofs_algo[2], uint8_t* algo_num,
+                              int *block_size_log2, int flag);
+
+
 /**
  * Inquire the number of zisofs compression and uncompression filters which
  * are in use.
@@ -7988,7 +8019,11 @@ int iso_zisofs_get_refcounts(off_t *ziso_count, off_t *osiz_count, int flag);
  */
 struct iso_zisofs_ctrl {
 
-    /* Set to 0 for this version of the structure */
+    /* Set to 0 or 1 for this version of the structure
+     * 0 = only members up to .block_size_log2 are valid
+     * 1 = members up to .max_file_blocks are valid
+     *     @since 1.5.4
+     */
     int version;
 
     /* Compression level for zlib function compress2(). From <zlib.h>:
@@ -7998,11 +8033,59 @@ struct iso_zisofs_ctrl {
      */
     int compression_level;
 
-    /* Log2 of the block size for compression filters. Allowed values are:
+    /* Log2 of the block size for compression filters of zisofs version 1.
+     * Allowed values are:
      *   15 = 32 kiB ,  16 = 64 kiB ,  17 = 128 kiB
      */
     uint8_t block_size_log2;
 
+    /* ------------------- Only valid with .version >= 1 ------------------- */
+
+    /* 
+     * @since 1.5.4
+     * Whether to produce zisofs2 (zisofs version 2) file headers and ZF
+     * entries for files which get compressed:
+     *  0 = do not produce zisofs2,
+     *      do not recognize zisofs2 file headers by magic
+     *      This is the default.
+     *  1 = zisofs2 is enabled for file size 4 GiB or more
+     *  2 = zisofs2 shall be used if zisofs is used at all
+     */
+    int v2_enabled;
+
+    /*
+     * @since 1.5.4
+     * Log2 of block size for zisofs2 files. 0 keeps current setting.
+     * Allowed are 15 = 32 kiB to 20 = 1024 kiB.
+     */
+    uint8_t v2_block_size_log2;
+
+    /*
+     * @since 1.5.4
+     * Maximum overall number of blocklist pointers. 0 keeps current setting.
+     */
+    uint64_t max_total_blocks;
+
+    /*
+     * @since 1.5.4
+     * Ignored as input value: Number of allocated zisofs block pointers.
+     */
+    uint64_t current_total_blocks;
+
+    /*
+     * @since 1.5.4
+     * Maximum number of blocklist pointers per file. 0 keeps current setting.
+     */
+    uint64_t max_file_blocks;
+
+    /* >>> ??? zisofs2: ISO_ZISOFS_MANY_BLOCKS , 0 = default 65537 */
+
+    /* >>> ??? zisofs2: a limit for number of zisofs2 files in order to keep
+                        the number of these old kernel warnings bearable:
+                          "isofs: Unknown ZF compression algorithm: PZ"
+                        0 = default >>> ??? value ? no limit ?
+    */
+
 };
 
 /**
@@ -8011,6 +8094,9 @@ struct iso_zisofs_ctrl {
  * i.e. ziso_count returned by iso_zisofs_get_refcounts() has to be 0.
  * @param params
  *      Pointer to a structure with the intended settings.
+ *      The caller sets params->version to indicate which set of members
+ *      has been filled. I.e. params->version == 0 causes all members after
+ *      params->block_size_log2 to be ignored.
  * @param flag
  *      Bitfield for control purposes, unused yet, submit 0
  * @return
@@ -8024,6 +8110,9 @@ int iso_zisofs_set_params(struct iso_zisofs_ctrl *params, int flag);
  * Get the current global parameters for zisofs filtering.
  * @param params
  *      Pointer to a caller provided structure which shall take the settings.
+ *      The caller sets params->version to indicate which set of members
+ *      shall be filled. I.e. params->version == 0 leaves all members after
+ *      params->block_size_log2 untouched.
  * @param flag
  *      Bitfield for control purposes, unused yet, submit 0
  * @return
@@ -8040,7 +8129,7 @@ int iso_zisofs_get_params(struct iso_zisofs_ctrl *params, int flag);
  * by an xinfo data record if not already marked by a zisofs compressor filter.
  * This does not install any filter but only a hint for image generation
  * that the already compressed files shall get written with zisofs ZF entries.
- * Use this if you insert the compressed reults of program mkzftree from disk
+ * Use this if you insert the compressed results of program mkzftree from disk
  * into the image.
  * @param node
  *      The node which shall be checked and eventually marked.
@@ -8756,7 +8845,7 @@ int iso_conv_name_chars(IsoWriteOpts *opts, char *name, size_t name_len,
 /** Use of zlib was not enabled at compile time (FAILURE, HIGH, -345) */
 #define ISO_ZLIB_NOT_ENABLED      0xE830FEA7
 
-/** Cannot apply zisofs filter to file >= 4 GiB  (FAILURE, HIGH, -346) */
+/** File too large. Cannot apply zisofs filter.  (FAILURE, HIGH, -346) */
 #define ISO_ZISOFS_TOO_LARGE      0xE830FEA6
 
 /** Filter input differs from previous run  (FAILURE, HIGH, -347) */
@@ -8765,7 +8854,7 @@ int iso_conv_name_chars(IsoWriteOpts *opts, char *name, size_t name_len,
 /** zlib compression/decompression error  (FAILURE, HIGH, -348) */
 #define ISO_ZLIB_COMPR_ERR        0xE830FEA4
 
-/** Input stream is not in zisofs format  (FAILURE, HIGH, -349) */
+/** Input stream is not in a supported zisofs format   (FAILURE, HIGH, -349) */