JBIG2 support added in UniPDF - golang

UniPDF is constantly improving itself and a while ago, we had achieved CCITT encoding, which allowed our users to better optimize PDFs. We’ve set another milestone and have integrated JBIG2 compression standard in our library. When it comes to the optimization of black and white images, JBIG2 is recognized as the best out there and offers a compression rate of 100:1.

Brief Description

JBIG2 is the standard for bi-level image compression, developed by the Joint Bi-level Image Experts Group. It is designed to compress black and white images in both lossless and lossy modes with better performance than the traditional JBIG and Fax Group 4 standards.

For PDF users, integration of JBIG2 means smaller file sizes without a loss to quality and readability. JBIG2 is highly effective when it comes to scanned documents and is the industry standard followed by acrobat while working with PDFs.

Compression Rate

As an example we would take a bi-level image of the checkerboard encoded in JPEG format with dimensions of 1920x1920 pixels and a size of 142,794 bytes. Illustration of bi-level image for JBIG2

Currently UniPDF’s JBIG2 encoder allows users to encode in a lossless mode. The checkerboard image compressed into the JBIG2 encoding format will only take 377 bytes, which can be achieved using the UniPDF’s JBIG2 encoder. This gives the compression ratio (uncompressed size/compressed size) of 378.76 times, which leads to 99.735% of optimized space savings the for the given example. Inserting images with JBIG2 Encoder filter into PDF files leads to a similar compression ratio of 160,121/1686 bytes, which is 94.97 times the original size and takes 98.947% less space.

Performance

The file size of a typical scanned document at 300dpi for a TIFF is around 75KB-125 KB per image. The same document encoded using JBIG2 would be about 5 to 10 times smaller (10KB - 15KB per image).

The performance also depends on the method used by the encoder. UniPDF allows users to encode black and white images using a generic lossless-encoding method, which is fast and has a relatively good compression ratio. For the future. We are focusing on building a proprietary lossy-encoding method. This would have a better compression ratio - especially for scanned text documents.

UniPDF JBIG2Encoder

JBIG2 standard allows to encode bi-level images in two modes:

  • lossless - the image quality is the same as original, no data is lost
  • lossy - better compression ratio, but some image parts might be lost

Lossless Generic region encoding

UniPDF library allows users to encode black and white images lossless-ly by providing a generic method. The encoder takes the whole image as a generic region and encodes it using the arithmetic coder. It reduces the file size by encoding the line duplicates using a single bit. This is used by setting DuplicateLinesRemoval. The more lines are duplicated the better the compression rate. This method is relatively fast with a basic compression.

Lossy - Classified - component encoding (Upcoming)

UniPDF is working on a classified component, lossy-encoding method. The lossy encoder would read and scan all pages of the provided document. The content of an image is decomposed into symbols and matched for similar occurrences. The symbols are stored in a Symbol Dictionary segment using an arithmetic coder stored at a given class index.

The encoder then takes all occurrences of the symbol classes and stores their position in a segment called ‘Text Region’. This encoding method has the best performance on images of text documents, i.e. scans. However, pure scans are imperfect in their quality and some letters may differ in a few bits representing a single pixel. Due to this, there is a correlation threshold parameter that allows the encoder to match ‘similar’ symbols even if they differ slightly. The value of that parameter is in range [0.0 - 1.0], where:

  • 0.0 - the symbols could be absolutely different in order to match (this value should never be used at all)
  • 1.0 - the symbols needs to be absolutely the same in order to match

For most scenarios, the threshold parameter in the range of 0.7 - 0.95 should provide us the best results. This parameter directly corresponds to the compression ratio - the lower the parameter the better the compression ratio. However, while lowering it’s value we should be very careful as this encoding method is lossy. The lower the threshold, the more lossy results we would obtain.

Global Symbols

JBIG2 Encoder allows you to store multiple black and white pages into a single JBIG2 document. This allows us to use an entity called ‘Global Symbols’, which acts as a single Symbol Dictionary for all encoded pages stored as separate byte streams.

Having a single store for the common symbols allows the encoder to reduce the size of the resulting JBIG2 byte stream for each page. It might be compared to the single, globally defined dictionary for the letters, where each page just takes a reference not the value by itself.

Global Symbols might only be used in the upcoming, lossy - classified method.

Examples

package main
import (
    "fmt"
    "image"
    "log"
    "path/filepath"
    // load jpeg decoder
    \_ "image/jpeg"
    "os"

    "github.com/unidoc/unipdf/v3/core"
)

func main() {
    // Let's read an jpeg rgba image from the file, convert it into JBIG2Image
    // using auto threshold and compress the black and white result into another file.
    f, err := os.Open("checkerboard-squares-black-white.jpg")
    if err != nil {
        log.Fatalf("Error: %v\n", err)
    }

    defer f.Close()

    img, \_, err := image.Decode(f)
    if err != nil {
        log.Fatalf("Error: %v\n", err)
    }

    // JBIG2Encoder requires core.JBIG2Image as an input.
    // In order to convert golang image.Image into core.JBIG2Image use
    // the core.GoImageToJBIG2 function.
    // For the RGB and gray scale images there is a 'threshold' which states
    // at which level the values should be black and at which it should be white.
    // It is recommended to use core.JB2ImageAutoThreshold value which computes image histogram.
    // and on it's base gets proper value for the threshold.
    // Convert the image into JBIG2Image using auto threshold.
    jb2Img, err := core.GoImageToJBIG2(img, core.JB2ImageAutoThreshold)
    if err != nil {
        log.Fatalf("Error: %v\n", err)
    }

    // Create a JBIG2 Encoder/Decoder context. In this example we're setting page settings
    // used for the encoding process.
    enc := &core.JBIG2Encoder{
    DefaultPageSettings: core.JBIG2EncoderSettings{
    // JBIG2 files could be stored as a separate files (mostly with .jb2 extension) or
    // as a part of PDF stream. In this case we want to store it as a file - thus set FileMode to true.
    FileMode: true,
    // In order to have better compression, JBIG2 encoder allows storing
    // duplicated lines (image row bits) once, and then relates to their value on the
    // subsequent rows.
    // In order to use it set DuplicatedLinesRemoval to true.
    DuplicatedLinesRemoval: true,
    },
}

    // Add JBIG2Image as a new page to the encoder context with the default page settings.
    if err = enc.AddPageImage(jb2Img, nil); err != nil {
        log.Fatalf("Error: %v\n", err)
    }

    // Encode the data into JBIG2 format and return as the slice of bytes.
    data, err := enc.Encode()
    if err != nil {
        log.Fatalf("Error: %v\n", err)
    }

    // Write encoded data into a file with the extension '.jb2' - this is standard extension for the JBIG2 files.
    encodedFile, err := os.Create("checkerboard-squares-black-white.jb2")
    if err != nil {
        log.Fatalf("Error: %v\n", err)
    }

    defer encodedFile.Close()
    \_, err = encodedFile.Write(data)
    if err != nil {
        log.Fatalf("Error: %v\n", err)
    }

    fmt.Println("Created JBIG2 Encoded file successfully")
}

Example - Encode image into PDF file

package main

import (
    "fmt"
    "log"
    "path/filepath"
    "os"

    "github.com/unidoc/unipdf/v3/common"
    "github.com/unidoc/unipdf/v3/core"
    "github.com/unidoc/unipdf/v3/creator"
)

func main() {
    // Read the image with the UniPDF creator.
    c := creator.New()
    img, err := c.NewImageFromFile("checkerboard-squares-black-white.jpg")
    if err != nil {
        log.Fatalf("Error loading image: %v", err)
    }

    // Convert the image into binary format. The RGB and GrayScale images would be converted into bi-level image.
    // This step is required for the JBIG2 Encoder.
    if err = img.ConvertToBinary(); err != nil {
        common.Log.Fatal("Err: %v", err)
    }

    // Set the JBIG2 Encoder as the image encoder.
    e := core.NewJBIG2Encoder()

    // For images that might equal following lines it might be convenient
    // to set DuplicatedLinesRemoval option to true.
    e.DefaultPageSettings.DuplicatedLinesRemoval = true
    img.SetEncoder(e)
    img.ScaleToWidth(612.0)

    // Use page width of 612 points, and calculate the height proportionally based on the image.
    // Standard PPI is 72 points per inch, thus a width of 8.5".
    height := 612.0 \* img.Height() / img.Width()
    c.SetPageSize(creator.PageSize{612, height})
    c.NewPage()
    img.SetPos(0, 0)

    if err = c.Draw(img); err != nil {
        common.Log.Fatal("Err: %v", err)
    }

    if err := c.WriteToFile("checkerboard-squares-black-white.pdf"); err != nil {
        common.Log.Fatal("Err: %v", err)
    }
    fmt.Println("Created JBIG2 Encoded file successfully")
}

UniPDF Converting Go Images to JBIG2Images

UniPDF JBIG2 Encoder accepts as an input only black and white images where the data is represented in a 1 bit per pixel (1bpp). The bits are written from Top Left corner defined row by row. Within a single byte, pixels from the left are located as the most significant bits. For images with width not divisible by 8 extra padding would occur on the last byte per row.

Unidoc UniPDF allows users to convert golang image.Image into core.JBIG2Image. It is done by the core.GoImageToJBIG2 function which takes two parameters:

  • image.Image - an input image to convert into binary JBIG2 format
  • Threshold - used for image conversion into Black and White pixels only. It might take the value in range [0.0 - 1.0]. This value represents how likely the value would take the white pixel value. For the special case when it’s value is 0.0 the converter computes the histogram of an image and adjusts the threshold value to it’s result.

Examples

package main
import (
    "os"
    "fmt"
    // read jpeg decoder
    "image/jpeg"
    "image"

    "github.com/unidoc/unipdf/v3/core"
)

func main() {
    // let's read an jpeg rgba image from the file
    // convert it into JBIG2Image using auto threshold
    // and store the black and white result into another file
    // read an image file 'my-image.jpg'
    f, err := os.Open("my-image.jpg")
    if err != nil {
        fmt.Printf("Error: %v\n", err)
        os.Exit(1)
    }

    defer f.Close()
    img, \_, err := image.Decode(f)

    if err != nil {
        fmt.Printf("Error: %v\n", err)
        os.Exit(1)
    }

    // convert the image into JBIG2Image
    jb2Img, err := core.GoImageToJBIG2(img, core.JB2ImageAutoThreshold)
    if err != nil {
        fmt.Printf("Error: %v\n", err)
        os.Exit(1)
    }

    // convert the JBIG2Image into golang image.Image
    bwImage, err := jb2Img.ToGoImage()
    if err != nil {
        fmt.Printf("Error: %v\n", err)
        os.Exit(1)
    }

    // create new image file for our black and white result.
    fBW, err := os.Create("my-image-bw.jpg")
    if err != nil {
        fmt.Printf("Error: %v\n", err)
        os.Exit(1)
    }
    defer fBW.Close()

    // encode the result into 'my-image-bw.jpg' file.
    err = jpeg.Encode(fBW, bwImage, &jpeg.Options{Quality: jpeg.DefaultQuality})
    if err != nil {
        fmt.Printf("Error: %v\n", err)
        os.Exit(1)
    }
}

UniPDF Supports All JBIG2 Decoding Formats

UniPDF golang library allows you to decode JBIG2 encoded files and byte streams. The library decodes byte streams implemented in all possible combinations using: MMR decoder, Arithmetic decoder and huffman tables decoder. It decodes all types of JBIG2 segments.

Decoder Example with Globals

package main

import (
    "os"
    "fmt"
    "image/jpeg"
    "io/ioutil"

    "github.com/unidoc/unipdf/v3/core"
)

func getByteStream(filename string) []byte {
    f, err := os.Open(filename)
    if err != nil {
        fmt.Printf("Error: %v\n", err)
        os.Exit(1)
    }

    defer f.Close()

    data, err := ioutil.ReadAll(f)
    if err != nil {
        fmt.Printf("Error: %v\n", err)
        os.Exit(1)
    }

    return data
}

func main() {
    // get the globals byte stream from some custom
    globalsData := getByteStream("globals.jb2")

    // create a separate encoding context for the globals
    globalEnc := &core.JBIG2Encoder{}
    globals, err := globalEnc.DecodeGlobals(globalsData)
    if err != nil {
        fmt.Printf("Error: %v\n", err)
        os.Exit(1)
    }

    // and the JBIG2 encoded data byte stream
     jbig2EncodedData := getByteStream("document.jb2")

     // create an encoder context for the 'JBIG2' data.
     enc := &core.JBIG2Encoder{
        // Set previously decoded JBIG2 globals in the context of target
        // encoder context.
         Globals: globals,
     }

     // we want to see the results, let's decode the byte stream into golang
     // image.Image JBIG2 documents may store more than one page - thus the result
     // is a slice of images
     imgs, err := enc.DecodeImages(jbig2EncodedData)
     if err != nil {
        fmt.Printf("Error: %v\n", err)
        os.Exit(1)
     }


    // Decode first page image.
    if len(imgs) > 1 {
        fmt.Printf("There should be only a single image decoded: %v\n", err)
        os.Exit(1)
    }


    // create a file to see the results.
    f, err := os.Create("decoded-img.jpg")
    if err != nil {
       fmt.Printf("Error: %v\n", err)
       os.Exit(1)
    }
    defer f.Close()

    // encode the result in a jpeg format
    if err = jpeg.Encode(f, imgs[0], &jpeg.Options{Quality: jpeg.DefaultQuality}); err != nil {
       fmt.Printf("Error: %v\n", err)
       os.Exit(1)
    }

}

Conclusion

The integration of JBIG2 encoding is the latest improvement in UniPDF and will allow our users to produce better optimized PDF documents. The introduction of JBIG2 also means that you can optimize your black and white scanned documents in a more efficient manner. This will help save storage without fearing the loss of quality. For the future, we are planning on developing a smart PDF compression system that will be able to identify whether the images are 1 bit or close to 1 bit and automatically compress them using JBIG2. We are developing the lossy JBIG2 encoding method, which would provide a better compression ratio. We are focused on building the most optimized PDF builder for our customers and will keep on perfecting UniPDF.

Acknowledgements

We gratefully acknowledge the following open source projects that served as references during development.

  • Apache Java PDFBox JBIG2 Decoder, Apache License 2.0. In order to achieve full support for the JBIG2 Decoder, it was necessary to implement all possible decoding combinations defined in the JBIG2 standard, aka ITU T.88 and ISO/IEC 14492. With a lack of Golang JBIG2 Open Source package, we’ve decided that it would be best to base our own implementation on some solid and reliable library. The Apache PDFBox JBIG2 library fulfilled all our requirements. It has a really good quality of the code along with the detailed comments on each function and class. It also implemented MMR, Huffman tables and arithmetic decompressors along with all JBIG2 segments.

  • AGL JBIG2 Encoder, Apache License 2.0. The complexity and lack of comprehensive documentation for the JBIG2 encoding process, lead us to look at the AGL JBIG2 Encoder library. At the moment of implementing our encoder it was the only Open Source JBIG2 encoder. It’s a C++ based library that implements both lossless and lossy encoding methods, where most of the image operations are done using DanBloomberg Leptonica library.

    The core encoding processes in the UniPDF JBIG2 Encoder were based on that well documented and solid library

  • DanBloomberg Leptonica, The 2-Clause BSD License, DanBloomberg Leptonica is an amazing C/C++ Open Source library. It provides raster operations, binary expansion and reduction, JBIG2 component creators, correlation scoring and a lot more perfectly commented image operation functions. That library was used as a very solid base for our image operation algorithms used by the JBIG2 Encoder.