5. General Reference Material

5.1 N64 JPEG Compression Process

The N64 JPEG compression method is based on the standard JPEG process. See the JPEG standard for a complete description of the exact process. Following is a brief explanation of the N64 JPEG compression process. The procedure for decompressing compressed data is basically the opposite of this.

5.1.1 Convert to the YUV Format

The N64 compression process first converts data from 16-bit RGBA format to YUV format. During this process, less essential visual data (data that is less important in human vision) is culled. The resulting YUV data format is easier to compress.

First the data is converted from RGB to YCrCb format. The Y value holds the luminance (brightness) information and Cr and Cb values hold the chrominance (color) information:

r = R/255, g = G/255, b = B/255
	(0<R<255, 0<G<255, 0<B<255)
	(0<r<1, 0<g<1, 0<b<1)

Y = 0.299r + 0.587g + 0.114b
Cr = 0.713(r - Y)
Cb = 0.564(b - Y)
	(0<Y<1, -0.5<Cr<0.5, -0.5<Cb<0.5)

Then the YCrCb formatted data is converted to the YUV format:

y = 219Y + 16
u = 224Cb + 128
v = 224Cr + 128
        (0<y<255, 0<u<255, 0<v<255)

5.1.2 Cull Less Essential Data

Human eyes are relatively insensitive to UV components compared to the Y component. Thus, culling the UV components can decrease the amount of data without a large noticeable change. This process extracts four 8x8 blocks of the Y component and one 8x8 block of each UV (U and V) component from the 16x16 dots (macro block).

5.1.3 DCT

DCT (discrete cosine transformation) regulates the apparently random image to ease the compression process.

When processing an 8x8 matrix, DCT calculates an average value (the DC coefficient) of the data elements in the matrix and places it in the [0][0] position of the transformed matrix. Then it calculates the variance (the AC coefficient) from that DC coefficient for each of the other 63 cells of the 8x8 matrix. The farther the AC element is from the position of the DC element, the higher the displayed frequency element is.

5.1.4 Quantization

The range of data after DCT is wide, and the wider the range the less compression, so you need to reduce the range. To reduce the range, divide each value by a specific value. Human vision is less sensitive to high frequency components than it is to low frequency components, so you should divide high frequency components by a large value and divide low frequency components by a small value.

5.1.5 Zigzag Scanning

The DCT and quantization processes concentrate on the DC components and low frequency components. The high frequency components don't change much and the data concentrates around 0. That is, large changes in data concentrate around cell [0][0] of the 8x8 pixel matrix after DCT, and many zeros concentrate around cell[7][7]. The continuous appearance of these zeros is profitable for compression, so this step rearranges the data to take advantage of the large concentration of zeros. One way to do this is to apply zigzag scanning to the data from [0][0] to [8][8] like this: [0][0] --> [1][0] --> [0][1] --> [0][2] --> [1][1] --> [2][1]... and so on.

DC components have differential data of DC components in the previous block so that the range of data values becomes smaller.

Note: The N64 JPEG process does the scanning after first inverting the matrix. This is different from normal JPEG zigzag scanning. It is done this way to ease microcode decompression of compressed data.

5.1.6 Entropy Encoding

The previous five steps converted the random image into a compressible form. Now this final step completes the Huffman entropy encoding to compress the image data.

5.2 Output Data Formats

5.2.1 Data from njpgMakeSlide Tool

njpgMakeSlideThe output data formats are described below.

When the output data from the tool is to be embeded with the game application, first DMA-transfer numImages of images onto the memory, and then DMA-transfer frame[] according to numImages after checking the necessary information. If image data is necessary, it should be DMA-transfered onto the memory as you reference the data pointed to by frame[].

typedef struct{
	s32 size;
	s32 offset;
} MVFrame;

struct{
	u8 headerID[] = 'NJPG'
	s32 version;
	s16 width;
	s16 height;
	u8 rawMode;
	u8 compressType;
	s16 qScale;
	s32 numImages;
	MVFrame frame[];
	u8 data[];
};

headerID

Recognition header of the N64 JPEG format data.

version

N64 JPEG version. The current version is "002".

width

Width of the image expressed in pixel (dot) units.

height

Height of the image expressed in pixel (dot) units.

rawMode

Raw data mode. It is 1 if the raw data mode is ON or 0 if it is OFF.

compressType

Entropy encoding method. A value of 0 indicates Huffman encoding.

numImages

Number of included images.

frame

Frame size and the offset address.

size Image size
offset The offset address for the image data.

data

The compressed image data.

5.2.2 Data from njpgCompress Tool

The data output by the njpgCompress tool is suited to only a part of the data output by the njpgMakeSlide tool. Be careful that the special supplemented header is not changed. Note that the minimum header required for data decompressing has been supplemented. The following is the data format. Refer to it to check the data when decompressing.

struct{
	u8	headerID[] = "HUFF";
	s16	numMB;		/* number of macro blocks */
	u8	compData[];
};

5.3 Quantization Table

The quantization table for both luminance (brightness) and chroma (color) is adapted to the luminosity quantization table of quantization table in the attachment K(reference) "Examples and guide line" of ITU-T advice, T.81 "The digital compression and coding of continuous-tone still images." The quantization table has been organized to make RSP calculation faster when decompressing the compressed data.

The table follows.

static short QTable[64] = {
	16, 11, 10, 16,  24,  40,  51,  61,
	12, 12, 14, 19,  26,  58,  60,  55,
	14, 13, 16, 24,  40,  57,  69,  56,
	14, 17, 22, 29,  51,  87,  80,  62,
	18, 22, 37, 56,  68, 109, 103,  77,
	24, 35, 55, 64,  81, 104, 113,  92,
	49, 64, 78, 87, 103, 121, 120, 101,
	72, 92, 95, 98, 112, 100, 103,  99
};

As a practical matter, this table is used by multiples of 1/4, 1/2, and 2 (these correspond to the scale values of -2, -1, 1, and 2). When the value of the quantization table is small, the image quality goes up but the size after compression is larger. On the other hand, when the value of the quantization table is large, the size after the compression is small but the image quality is worse.

5.4 Huffman Encoding Table

The Huffman encoding table is the one in the attachment K (reference) "Examples and Guidelines" of the ITU-T advice, T.81 "Digital compression and encoding of continuous-tone still images"

These tables follow.

Normally with JPEG compression process using the Huffman encoding to output the data, when an identical code (0xFF) with the marker appers in the data, the code should be followed by "0x00" by inserting it to indicate that the code is not a marker. This is because an ordinary data should be distinguished from a marker that defines information regarding the image. However, this process is not included because in N64 JPEG doesn't use a marker.

Table of DC Codes for Luminance (Brightness)

Table of DC Codes for Chroma (Color)

Table of AC Codes for Luminance (Brightness)

Table of AC Codes for Chroma (Color)

5.5 Programming Cautions

5.5.1 Image Quality

N64 JPEG raises the compression rate by slightly reducing the image quality. Therefore, if you repeatedly compress and decompress the compressed image, image quality can become noticeably worse.

Also, the N64 JPEG process includes errors, so you must be particularly careful when culling the UV data in the YUV space. For example, if you make an N64 drawing tool for use over and over in the N64 JPEG library for the purpose of image storage and retrieval, image encoding and decoding are provided whenever you repeat the storage and retrieval process. This means that if the drawing is done in the RGBA space (in the regular N64 image process space), UV data in the macro block is culled again and again based on the drawing. In short, an application that repeatedly compresses and decompresses a compressed image provides data culling over and over which can cause a noticeable deterioration in image quality.

If you make such an application, you should store the regular RGBA data without using compression or store the compressed data (for example, by the run length encoding as in the compression used in a BMP file), and use N64 JPEG as the final storage method.

5.5.2 Data Form After Decompression

Data decoded by the njpgdspMain microcode is output in YUV format, not RGBA format. Therefore, you need to be aware that the image won't be displayed normally, even if you directly specify the output data buffer to the frame buffer.

Also, data output in YUV format needs to be expressed as a 16x16 dot texture. Both input and output data regions are divided into 768-byte macro blocks, and each macro block processed completes in its region. As a result, the size of the output data becomes 512 byte for the 768 byte (Y4, U1, and V1 are each 128 bytes) after input to the macro block region. The 256-byte empty region (suitable for the unnecessary UV component region) is produced. (Refer to "3.1.3 Output Data")

When you get the texture from the data buffer, add the pointer to the buffer at each 768-byte boundary.

Please refer to Sample Program for the specific methods.