Compression Schemes for High Dimensional Data based on Extendible Multidimensional Arrays

Abstract

Traditional Multidimensional Array (TMA) is an important data structure for handling large scale multidimensional dataset, but they are not extendible during run time. Another problem for representing the real life data by multidimensional arrays is that it creates high degree of sparsity. Due to this sparsity problem and increasing size of the data structures, it becomes necessity to develop a suitable scheme to compress the multidimensional array in an efficient way so that it takes comparatively low memory storage. To minimize both of these sparsity and reorganization problem novel schemes are proposed to compress high dimensional data based on dynamically extendible array. In this research work we propose compression schemes based on Extendible multidimensional array. The proposed compression schemes are Extendible array based Compressed Row Storage (EaCRS) scheme, Linearized Extendible array based Compressed Row Storage (LEaCRS) scheme and Extendible array based Chunk Offset Compression Scheme (EaChOfJ. The main idea of both the EaCRS and LEaCRS scheme is to compress the subarrays independently found from the existing extendible array. LEaCRS scheme differs from EaCRS scheme only in the way that the LEaCRS scheme needs to linearize each subarray first and then compresses the subarray independently. EaChOJj scheme linearizes each subarray independently and breaks a large multi dimensional extendible array into chunks for compressing. In this scheme, a maximum size of each chunk is considered and chunks are formed by one or more subarrays. We evaluated our proposed schemes by comparing compression ratio, data retrieval time and extension cost with CR3 on TMA and ChunkOJjei Compression on TMA. Both analytical analysis and experimental tests were conducted. The analytical analysis and experimental results show that the proposed schemes have better range of usability and compression ratio for practical applications than traditional schemes. Furthermore, we found that the retrieval time of the proposed compression schemes are independent of different dimensions. The increment operation will be efficient in the proposed compression schemes than the existing traditional compression schemes because it increments without reorganizing the previous data.