BWT Backward Search Author: Magnus Hagmar ZID: z5088131 The provided code implements BWT backward searching, as proposed by Paolo Ferragina and Giovanni Manzini, in a BWT encoded record file. The main bottleneck of BWT backward search is counting the number of previous occurrences of a character in the file. The number of calls to the occurrence function (occ) is linear in the length of the search pattern when searching for any matching occurrences. In addition to this, one call to occ is required for every single step when backtracking from a result to determine which record it belongs to. This could result in hundreds or even thousands of calls. Clearly, a naive implementation of counting every character up to the desired point is not plausible. This implementation generates an index file to speed up the occ function. This file is created by reading characters from the input file while keeping count of how many of each have occurred. Every time 392 (this seemingly random number will be explained later) characters have been read, the current occurrence counts for all characters up to that point are written to the index file as a block. By utilizing an index constructed in this way, the number of characters which have to be read from the input file can be minimized. For example, if counting the number of 'A's which have occurred up until index 800, we could use the index to extract the number of 'A's up until index 2*392=784 and then simply check the remaining 16 characters to see if they are also 'A'. This is significantly faster than reading and observing all 800 characters up to that point. The choice to read 392 characters into each block is not random, but rather the optimal choice considering the specifications and limitations of the task. First of all, there are only 98 characters which can occurr in the text file (tab, newline, carriage return and the visible ASCII characters 32-126). As such, the size of a block of occurrences written to disk is 98*4=392 bytes. The four comes from using an unsigned int to store the occurrences of each character. There is also a limitation that the index file is not permitted to exceed the original file in size. So if a block requires 392 bytes of disk space, then we need to include 392 bytes from the original file in each block. If less than 392 characters were counted for each block, then the index file would grow larger than the original file. It is possible to count more than 392 characters into each block, which would result in a smaller index file, but that would also result in a less accurate index and more characters having to be counted when running the occ function. Description of the different files: - index.h & index.cpp These files implement a an Index class, managing all usage of the occurrence index and C-table required for performing BWT backward search. Since the occurrence index only keeps track of 98 characters, a mapping is required from each character to its corresponding index in the 98 elements. An array "charMap" is used for this, indexed simply by the characters' numerical value and mapping to their new index in the compressed array. Creation of the index file as described above is managed by the Index class. The C-table is created by fetching the last occurrence block and then counting any remaining characters which haven't been included in the block. In addition to this, the C-table contains an extra entry after ASCII character 126. This is due to being required to look up C[c+1] in the backward search algorithm. Without this additional dummy character, this lookup would fail when c is '~', ASCII number 126. The charMap described above also contains dummy entries to prevent these failures. - bwtsearch.h & bwtsearch.cpp These files implement the main function as well as the actual backward search algorithm and functions for interpreting the search results. The backward search is identical to Ferragina's and Manzini's algorithm, except for the detail that all indexing is zero-based to simplify indexing in arrays. The findRecords function implements a BWT decoder to step backwards through the original file in order to find the record to which a search result belongs to. This set of records is then either counted or printed, depending on the mode of searching.