Parsing file format for attributes in Raster Product Format (RPF) maps and imagery?

Question

I'm trying to implement an open-source NITF reader (using only java, no external libraries such as GDAL or NITRO), and its mostly going OK. However I'm having trouble understanding the RPF extensions, in particular the attributes in the RPFDES TRE.

I know that they are documented in MIL-STD-2411-2, and I've read that and the main references (MIL-STD-2411 and MIL-STD-2411-1). I still don't understand how to parse it out though.

As a concrete example, http://download.osgeo.org/gdal/data/nitf/bugs/bug3337.ntf (which is NITF 2.0) has the RPFDES TRE looking like:

01efe0 20 02 00 20 02 00 20 02 00 20 02 44 45 52 65 67  > .. .. .. .DEReg<
01eff0 69 73 74 65 72 65 64 20 45 78 74 65 6e 73 69 6f  >istered Extensio<
01f000 6e 73 20 20 20 20 30 31 55 20 20 20 20 20 20 20  >ns    01U       <
01f010 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20  >                <
*
01f0a0 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 55  >               U<
01f0b0 44 49 44 20 20 30 30 31 30 30 30 30 52 50 46 44  >DID  0010000RPFD<
01f0c0 45 53 30 30 36 32 31 00 21 00 02 00 00 00 00 00  >ES00621.!.......<
01f0d0 08 00 01 01 00 00 00 01 08 00 02 01 00 00 00 01  >................<
01f0e0 10 00 03 01 02 00 00 01 18 00 04 01 02 00 00 01  >................<
01f0f0 20 00 04 02 02 00 00 01 2a 00 04 03 02 00 00 01  > .......*.......<
01f100 32 00 04 04 02 00 00 01 36 00 05 01 02 00 00 01  >2.......6.......<
01f110 3d 00 05 02 02 00 00 01 3f 00 05 03 02 00 00 01  >=.......?.......<
01f120 43 00 05 04 02 00 00 01 47 00 06 01 01 00 00 01  >C.......G.......<
01f130 4b 00 07 01 01 00 00 01 4f 00 08 02 01 00 00 01  >K.......O.......<
01f140 53 00 09 02 01 00 00 01 55 00 0a 02 01 00 00 01  >S.......U.......<
01f150 57 00 0b 02 01 00 00 01 59 00 0c 01 02 00 00 01  >W.......Y.......<
01f160 5b 00 0e 01 02 00 00 01 5e 00 10 01 02 00 00 01  >[.......^.......<
01f170 60 00 10 02 02 00 00 01 64 00 11 01 02 00 00 01  >`.......d.......<
01f180 66 00 11 02 02 00 00 01 6a 00 12 01 02 00 00 01  >f.......j.......<
01f190 6c 00 12 02 02 00 00 01 70 00 13 01 02 00 00 01  >l.......p.......<
01f1a0 72 00 13 02 02 00 00 01 76 00 14 01 02 00 00 01  >r.......v.......<
01f1b0 78 00 14 02 02 00 00 01 80 00 14 03 02 00 00 01  >x...............<
01f1c0 82 00 14 04 02 00 00 01 8a 00 19 01 02 00 00 01  >................<
01f1d0 92 00 19 02 02 00 00 01 94 31 39 38 34 30 33 31  >.........1984031<
01f1e0 30 32 30 30 31 31 30 31 32 31 39 38 34 30 33 31  >0200110121984031<
01f1f0 30 47 4e 43 20 20 20 20 20 20 20 30 31 20 20 20  >0GNC       01   <
01f200 20 20 20 55 4e 44 20 20 20 20 20 20 20 37 54 43  >   UND       7TC<
01f210 00 00 00 00 40 40 00 00 3f 7f e5 c9 4d 53 4c 20  >....@@..?...MSL <
01f220 57 47 45 00 00 00 00 00 00 00 00 00 49 4e 54 75  >WGE.........INTu<
01f230 64 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  >d...............<
01f240 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00  >................<
*
01f260 00 00 00 00 00 00 00 00 00 00 08 00 52 00 10 00  >............R...<
01f270 05 40 52 f6 db ba 46 e8 05 c0 60 e0 00 00 00 00  >[email protected]...`.....<
01f280 00 40 52 f6 db ba 46 e8 05 c0 60 e0 00 00 00 00  >[email protected]...`.....<
01f290 00 40 52 f6 db ba 46 e8 05 40 46 80 00 00 00 00  >[email protected]..@F.....<
*
01f2b0 00 40 52 f6 db ba 46 e8 05 c0 60 e0 00 00 00 00  >[email protected]...`.....<
01f2c0 00 00 05 40 56 80 00 00 00 00 00 c0 66 80 00 00  >[email protected]...<
01f2d0 00 00 00 40 56 80 00 00 00 00 00 40 66 80 00 00  >...@V......@f...<
01f2e0 00 00 00 40 50 ff f6 ab 93 ae fe 40 66 80 00 00  >...@P......@f...<
01f2f0 00 00 00 40 50 ff f6 ab 93 ae fe c0 66 80 00 00  >[email protected]...<
01f300 00 00 00 40 56 80 00 00 00 00 00 c0 66 80 00 00  >[email protected]...<
01f310 00 00 00 00 00 00 08 00 01 00 19 30 30 30 30 30  >...........00000<
01f320 30 32 33 2e 47 4e 39 30 30 30 30 30 30 31 33 2e  >023.GN900000013.<
01f330 47 4e 39 33                                      >GN93<
01f334

I'm OK with the parts up and including the RPFDES tag name and length. After 01f0c7, I'm lost.

From using GDAL, I can see that there are some dates starting at 0x01f1d9, and some other things that are probably the CRS and elevation references, but I'm mystified as to how I'm meant to extract them. I have no problem actually reading data in java (I've got most of NITF implemented), just don't understand which data I need to read. Can someone explain the data structure parsing to get to the first date, or at least provide some additional guidance on how I'm meant to use MIL-STD-2411-2 to work it out?

For those wondering why it has to be only java - its a deployment thing for OSGI. For those wondering where the source code is, see https://github.com/codice/imaging-nitf.

Answer 1

This isn't 100% complete, but I think should be enough to help get you on the right track. The super confusing part here is that you need to reference a ton of different standards, each of which has multiple versions. Hope this helps!

MIL-STD-2500A

First, we know the file follows this specific version of the standard because it begins with NITF02.00 (if it followed 2500B/C this would be NITF02.10) TABLE XVII explains the fields included in the "Data Extension Segment" (DES) which starts at 0x01efeb. That table is enough to explain everything up until 0x01f0bc where the DESDATA field starts.

MIL-STD-2411-2

Section 5.3.3 explains the structure of DESDATA, which in the context of this document is referred to as the [nitf user-defined data extension group]. After the 6-byte ASCII <nitf-tag> and the 5-byte <nitf tagged data subgroup length> fields we should expect to see one or more [rpf component] sections.

That brings us up to 0x01f0c7, where your question really begins. The main problem is that 2411-2 is pretty vague about what exactly these [rpf component] sections contain. Section 4.1(c) explains:

Mot: of the names assigned herein to rpf groups (i.e. those beginning with che prefix [rpf ...] correspond exactly to those assigned to those groups in MIL-STD-2411. ... The names that are not found in MIL-STD-2411 (e.g. [rpf sections and components] are generic names that are used herein only.

Luckily the example in Appendix 20 (lines 90 and 91) shows that (in the case of frame files, at least) there should be two groups: [rpf attribute section] followed by the [rpf replace/update section], which are defined for us in 2411.

MIL-STD-2411

Figure 3 provides a detailed description of 'frame file' structure, including the [attribute section] mentioned above. It contains two main subsections:

• The [attribute subsection] contains various metadata attributes. The actual attribute values are all concatenated together into one big chunk of bytes called the [attribute table]. This is preceded by the [attribute offset table], which lists the IDs of the various attributes (as described in MIL-STD-2411-1) and "offsets" that help locate each value within the attribute table.
• The [explicit areal coverage subsection] describes the image footprint (I'll use the acronym EAC for brevity)

First, however, is the [attribute section subheader], which is 10 bytes long:

$ xxd -l 10 -s +0x01f0c7 bug3337.ntf
0001f0c7: 0021 0002 0000 0000 0008                 .!........

• The first two bytes are the number of attribute records (33)
• The next two bytes are the number of EAC records (2)
• The next four bytes are the number of 'blank' bytes which follow the header (0)
• The last two bytes is the length of each 'attribute offset record' (8)

Based on the information in this subheader, we can conclude that the [attribute offset table] should be 33*8=264 bytes long. Therefore, we should expect to see the first byte of the [attribute table] itself at 0x1f1d9 (where you note that you see some dates). First, though, let's look at the first <attribute offset record> in the [attribute offset table], which begins immediately after the subheader and is 8 bytes long.

$ xxd -l 8 -s +0x01f0d1 bug3337.ntf
0001f0d1: 0001 0100 0000 0108                      ........

• The first two bytes are the attribute ID (1)
• The next byte is the parameter ID (also 1))
• The next byte is the areal coverage sequence number (0)
• The last 4 bytes is the offset (relative to beginning of the [attribute subsection]) where we can expect to find the value for this attribute (0x108).

If we look up the Attribute and Parameter ID's in 2411-1, we see that this represents the Currency Date, and we should expect the value to be 8 bytes of ASCII. Additionally, the offset of 0x108 bytes is exactly the same as the 264 bytes we calculated above from the subheader, so we know we aren't totally lost yet!

We can inspect the actual value, and sure enough it looks like a date:

$ xxd -l 8 -s +0x1f1d9 bug3337.ntf
0001f1d9: 3139 3834 3033 3130                      19840310

We can get the entire attribute offset table like so:

$ xxd -l 264 -s +0x01f0d1 bug3337.ntf
0001f0d1: 0001 0100 0000 0108 0002 0100 0000 0110  ................
0001f0e1: 0003 0102 0000 0118 0004 0102 0000 0120  ...............
0001f0f1: 0004 0202 0000 012a 0004 0302 0000 0132  .......*.......2
0001f101: 0004 0402 0000 0136 0005 0102 0000 013d  .......6.......=
0001f111: 0005 0202 0000 013f 0005 0302 0000 0143  .......?.......C
0001f121: 0005 0402 0000 0147 0006 0101 0000 014b  .......G.......K
0001f131: 0007 0101 0000 014f 0008 0201 0000 0153  .......O.......S
0001f141: 0009 0201 0000 0155 000a 0201 0000 0157  .......U.......W
0001f151: 000b 0201 0000 0159 000c 0102 0000 015b  .......Y.......[
0001f161: 000e 0102 0000 015e 0010 0102 0000 0160  .......^.......`
0001f171: 0010 0202 0000 0164 0011 0102 0000 0166  .......d.......f
0001f181: 0011 0202 0000 016a 0012 0102 0000 016c  .......j.......l
0001f191: 0012 0202 0000 0170 0013 0102 0000 0172  .......p.......r
0001f1a1: 0013 0202 0000 0176 0014 0102 0000 0178  .......v.......x
0001f1b1: 0014 0202 0000 0180 0014 0302 0000 0182  ................
0001f1c1: 0014 0402 0000 018a 0019 0102 0000 0192  ................
0001f1d1: 0019 0202 0000 0194                      ........

I won't go through each attribute, but looking at the final 8 bytes we can see that the last record in the attribute table should start at 0x1f0d1 + 0x194 = 0x1f265. The Attribute ID is 25 and the Parameter ID is 2. Looking at 2411-1, we can see that the value should be a 2 bytes integer representing the unit of measurement for the contour interval (note that this is only included in 2411-1-Update-1). In this case the value is 0, which indicates unknown units. The value doesn't really matter, though, the main thing is that we now know that the [explicit areal coverage subsection] should begin at 0x1f265 + 0x2 = 0x1f267.

Let's inspect the first 8 bytes of the EAC subsection. The standard doesn't refer to these as a "subheader", but they feel like one:

$ xxd -l 8 -s +0x1f267 bug3337.ntf
0001f267: 0000 0008 0052 0010                      .....R..

• The first 4 bytes indicate the offset from the beggining of the (EAC) subsection to the start of the EAC table (8). In this case, the EAC table starts immediately after this "subheader"
• The next two bytes indicate the length of each EAC record (82)
• The last two bytes indicate the length of the "corner coordinates record" (16)

Since we above that we should expect two EAC records, this section should end at 0x1f267 + 0x8 + 0xac = 0x1f313. This takes us almost all the way to the end of the file.