orcfile

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.0.2/ds_Hive/orcfile.html

Compared with RCFile format, for example, ORC file format has many advantages such as:

• a single file as the output of each task, which reduces the NameNode's load
• Hive type support including datetime, decimal, and the complex types (struct, list, map, and union)（复杂数据类型）
• light-weight indexes stored within the file
  • skip row groups that don't pass predicate filtering
  • seek to a given row
• block-mode compression based on data type
  • run-length encoding for integer columns（RLE针对整数压缩效果更好）
  • dictionary encoding for string columns
• concurrent reads of the same file using separate RecordReaders
• ability to split files without scanning for markers（不使用同步标记）
• bound the amount of memory needed for reading or writing
• metadata stored using Protocol Buffers, which allows addition and removal of fields

---

An ORC file contains groups of row data called stripes, along with auxiliary information in a file footer.

• At the end of the file a postscript holds compression parameters and the size of the compressed footer.
• The default stripe size is 250 MB. Large stripe sizes enable large, efficient reads from HDFS.
• The file footer contains a list of stripes in the file, the number of rows per stripe, and each column's data type. It also contains column-level aggregates count, min, max, and sum.

#note: 文件格式和leveldb SSTable非常类似

As shown in the diagram, each stripe in an ORC file holds index data, row data, and a stripe footer.

• The stripe footer contains a directory of stream locations. Row data is used in table scans. (和 RCFile 相比row-data这个部分应该类似)
• Index data includes min and max values for each column and the row positions within each column (A bit field or bloom filter could also be included.) Row index entries provide offsets that enable seeking to the right compression block and byte within a decompressed block. (和 RCFile 相比，index-data应该对应的是meta-data部分。index-data包括了概要信息以及bloom filter可以用来skip整个stripe. index-data的row-position是基于compression block偏移，所以不用像rcfile一样需要解压整个column, 这样读取量会减少不少)
• Having relatively frequent row index entries enables row-skipping within a stripe for rapid reads, despite large stripe sizes. By default every 10,000 rows can be skipped.
• With the ability to skip large sets of rows based on filter predicates, you can sort a table on its secondary keys to achieve a big reduction in execution time. For example, if the primary partition is transaction date, the table can be sorted on state, zip code, and last name. Then looking for records in one state will skip the records of all other states.(可以选择性地针对column做排序)