com.orientechnologies.orient.core.storage.cache.local

Class OWOWCache

java.lang.Object

com.orientechnologies.orient.core.storage.cache.OAbstractWriteCache

com.orientechnologies.orient.core.storage.cache.local.OWOWCache

All Implemented Interfaces:

OCachePointer.WritersListener, OWriteCache

public final class OWOWCache
extends OAbstractWriteCache
implements OWriteCache, OCachePointer.WritersListener

Write part of disk cache which is used to collect pages which were changed on read cache and store them to the disk in background thread. In current implementation only single background thread is used to store all changed data, despite of SSD parallelization capabilities we suppose that better to write data in single big chunk by one thread than by many small chunks from many threads introducing contention and multi threading overhead. Another reasons for usage of only one thread are

1. That we should give room for readers to read data during data write phase
2. It provides much less synchronization overhead

Background thread is running by with predefined intervals. Such approach allows SSD GC to use pauses to make some clean up of half empty erase blocks. Also write cache is used for checking of free space left on disk and putting of database in "read mode" if space limit is reached and to perform fuzzy checkpoints. Write cache holds two different type of pages, pages which are shared with read cache and pages which belong only to write cache (so called exclusive pages). Files in write cache are accessed by id , there are two types of ids, internal used inside of write cache and external used outside of write cache. Presence of two types of ids is caused by the fact that read cache is global across all storages but each storage has its own write cache. So all ids of files should be global across whole read cache. External id is created from internal id by prefixing of internal id (in byte presentation) with bytes of write cache id which is unique across all storages opened inside of single JVM. Write cache accepts external ids as file ids and converts them to internal ids inside of its methods.

Since:

7/23/13

Author:

Andrey Lomakin (a.lomakin-at-orientdb.com)

Field Summary

Fields

Modifier and Type	Field and Description
static int	CHECKSUM_OFFSET
static long	MAGIC_NUMBER_WITH_CHECKSUM Marks pages which have a checksum stored.
static long	MAGIC_NUMBER_WITH_CHECKSUM_ENCRYPTED Marks pages which have a checksum stored and data encrypted

Constructor Summary

Constructors

Constructor and Description

OWOWCache(int pageSize, OByteBufferPool bufferPool, OWriteAheadLog writeAheadLog, DoubleWriteLog doubleWriteLog, long pagesFlushInterval, int shutdownTimeout, long exclusiveWriteCacheMaxSize, Path storagePath, String storageName, OBinarySerializer<String> stringSerializer, OClosableLinkedContainer<Long,OFile> files, int id, OChecksumMode checksumMode, byte[] iv, byte[] aesKey, boolean callFsync, boolean useNativeOsAPI)

Method Summary

Modifier and Type	Method and Description
void	addBackgroundExceptionListener(OBackgroundExceptionListener listener) Adds listener which is triggered if exception is cast inside background flush data thread.
long	addFile(String fileName)
long	addFile(String fileName, long fileId)
void	addLowDiskSpaceListener(OLowDiskSpaceListener listener)
void	addOnlyWriters(long fileId, long pageIndex)
void	addPageIsBrokenListener(OPageIsBrokenListener listener) Adds listener which is called by cache if corruption of file page is detected.
int	allocateNewPage(long fileId)
long	bookFileId(String fileName)
void	checkCacheOverflow()
boolean	checkLowDiskSpace()
OPageDataVerificationError[]	checkStoredPages(OCommandOutputListener commandOutputListener)
long[]	close()
void	close(long fileId, boolean flush)
void	create()
long[]	delete()
void	deleteFile(long fileId)
boolean	exists(long fileId)
boolean	exists(String fileName)
long	externalFileId(int fileId) Converts unique internal file id to external one.
long	fileIdByName(String fileName) Returns id associated with given file or value < 0 if such file does not exist.
boolean	fileIdsAreEqual(long firsId, long secondId) DO NOT DELETE THIS METHOD IT IS USED IN ENTERPRISE STORAGE
String	fileNameById(long fileId)
Map<String,Long>	files()
void	flush()
void	flush(long fileId)
void	flushTillSegment(long segmentId)
long	getCacheOverflowCount()
long	getExclusiveWriteCachePagesSize()
long	getExclusiveWriteCacheSize()
long	getFilledUpTo(long fileId)
int	getId()
Long	getMinimalNotFlushedSegment()
Path	getRootDirectory() Directory which contains all files managed by write cache.
long	getWriteCacheSize()
int	internalFileId(long fileId) Returns internal file id which is unique and always the same for given file in contrary to external id which changes over close/open cycle of cache.
OCachePointer	load(long fileId, long startPageIndex, OModifiableBoolean cacheHit, boolean verifyChecksums)
long	loadFile(String fileName) Registers new file in write cache and returns file id assigned to this file.
void	loadRegisteredFiles() Loads files already registered in storage.
void	makeFuzzyCheckpoint(long segmentId, byte[] lastMetadata)
String	nativeFileNameById(long fileId) Obtains native file name by the given file id.
void	open()
int	pageSize() DO NOT DELETE THIS METHOD IT IS USED IN ENTERPRISE STORAGE
void	removeBackgroundExceptionListener(OBackgroundExceptionListener listener) Removes listener which is triggered if exception is cast inside background flush data thread.
void	removeLowDiskSpaceListener(OLowDiskSpaceListener listener)
void	removeOnlyWriters(long fileId, long pageIndex)
void	removePageIsBrokenListener(OPageIsBrokenListener listener) Removes listener which is called by cache if corruption of file page is detected.
void	renameFile(long fileId, String newFileName)
String	restoreFileById(long fileId) Finds if there was file in write cache with given id which is deleted right now.
void	restoreModeOff()
void	restoreModeOn()
void	setChecksumMode(OChecksumMode checksumMode)
void	store(long fileId, long pageIndex, OCachePointer dataPointer)
void	truncateFile(long fileId)
void	updateDirtyPagesTable(OCachePointer pointer, OLogSequenceNumber startLSN)

Methods inherited from class com.orientechnologies.orient.core.storage.cache.OAbstractWriteCache

checkFileIdCompatibility, composeFileId, extractFileId, extractStorageId

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

MAGIC_NUMBER_WITH_CHECKSUM

public static final long MAGIC_NUMBER_WITH_CHECKSUM

Marks pages which have a checksum stored.

See Also:

Constant Field Values

MAGIC_NUMBER_WITH_CHECKSUM_ENCRYPTED

public static final long MAGIC_NUMBER_WITH_CHECKSUM_ENCRYPTED

Marks pages which have a checksum stored and data encrypted

See Also:

Constant Field Values

CHECKSUM_OFFSET

public static final int CHECKSUM_OFFSET

See Also:

Constant Field Values

Constructor Detail

OWOWCache

public OWOWCache(int pageSize,
                 OByteBufferPool bufferPool,
                 OWriteAheadLog writeAheadLog,
                 DoubleWriteLog doubleWriteLog,
                 long pagesFlushInterval,
                 int shutdownTimeout,
                 long exclusiveWriteCacheMaxSize,
                 Path storagePath,
                 String storageName,
                 OBinarySerializer<String> stringSerializer,
                 OClosableLinkedContainer<Long,OFile> files,
                 int id,
                 OChecksumMode checksumMode,
                 byte[] iv,
                 byte[] aesKey,
                 boolean callFsync,
                 boolean useNativeOsAPI)

Method Detail

loadRegisteredFiles

public void loadRegisteredFiles()
                         throws IOException,
                                InterruptedException

Loads files already registered in storage. Has to be called before usage of this cache

Throws:

IOException

InterruptedException

addBackgroundExceptionListener

public void addBackgroundExceptionListener(OBackgroundExceptionListener listener)

Adds listener which is triggered if exception is cast inside background flush data thread.

Specified by:

addBackgroundExceptionListener in interface OWriteCache

Parameters:

listener - Listener to trigger

removeBackgroundExceptionListener

public void removeBackgroundExceptionListener(OBackgroundExceptionListener listener)

Removes listener which is triggered if exception is cast inside background flush data thread.

Specified by:

removeBackgroundExceptionListener in interface OWriteCache

Parameters:

listener - Listener to remove

getRootDirectory

public Path getRootDirectory()

Directory which contains all files managed by write cache.

Specified by:

getRootDirectory in interface OWriteCache

Returns:

Directory which contains all files managed by write cache or null in case of in memory database.

addLowDiskSpaceListener

public void addLowDiskSpaceListener(OLowDiskSpaceListener listener)

Specified by:

addLowDiskSpaceListener in interface OWriteCache

addPageIsBrokenListener

public void addPageIsBrokenListener(OPageIsBrokenListener listener)

Description copied from interface: OWriteCache

Adds listener which is called by cache if corruption of file page is detected.

Specified by:

addPageIsBrokenListener in interface OWriteCache

removePageIsBrokenListener

public void removePageIsBrokenListener(OPageIsBrokenListener listener)

Description copied from interface: OWriteCache

Removes listener which is called by cache if corruption of file page is detected.

Specified by:

removePageIsBrokenListener in interface OWriteCache

removeLowDiskSpaceListener

public void removeLowDiskSpaceListener(OLowDiskSpaceListener listener)

Specified by:

removeLowDiskSpaceListener in interface OWriteCache

bookFileId

public long bookFileId(String fileName)

Specified by:

bookFileId in interface OWriteCache

pageSize

public int pageSize()

Description copied from interface: OWriteCache

DO NOT DELETE THIS METHOD IT IS USED IN ENTERPRISE STORAGE

Specified by:

pageSize in interface OWriteCache

Returns:

Size of page inside of cache.

fileIdsAreEqual

public boolean fileIdsAreEqual(long firsId,
                               long secondId)

Description copied from interface: OWriteCache

DO NOT DELETE THIS METHOD IT IS USED IN ENTERPRISE STORAGE

Takes two ids and checks whether they are equal from point of view of write cache. In other words methods checks whether two ids in reality contain the same internal ids.

Specified by:

fileIdsAreEqual in interface OWriteCache

loadFile

public long loadFile(String fileName)
              throws IOException

Description copied from interface: OWriteCache

Registers new file in write cache and returns file id assigned to this file.

File id consist of two parts:

1. Internal id is permanent and can not be changed during life of storage OWriteCache.internalFileId(long)
2. Write cache id which is changed between storage open/close cycles

If file with the same name is deleted and then new file is created this file with have the same internal id.

Specified by:

loadFile in interface OWriteCache

Parameters:

fileName - Name of file to register inside storage.

Returns:

Id of registered file

Throws:

IOException

addFile

public long addFile(String fileName)
             throws IOException

Specified by:

addFile in interface OWriteCache

Throws:

IOException

fileIdByName

public long fileIdByName(String fileName)

Description copied from interface: OWriteCache

Returns id associated with given file or value < 0 if such file does not exist.

Specified by:

fileIdByName in interface OWriteCache

Parameters:

fileName - File name id of which has to be returned.

Returns:

id associated with given file or value < 0 if such file does not exist.

internalFileId

public int internalFileId(long fileId)

Description copied from interface: OWriteCache

Returns internal file id which is unique and always the same for given file in contrary to external id which changes over close/open cycle of cache.

Specified by:

internalFileId in interface OWriteCache

Parameters:

fileId - External file id.

Returns:

Internal file id.

externalFileId

public long externalFileId(int fileId)

Description copied from interface: OWriteCache

Converts unique internal file id to external one. External id is combination of internal id and write cache id, which changes every time when cache is closed and opened again.

Specified by:

externalFileId in interface OWriteCache

Parameters:

fileId - Internal file id.

Returns:

External file id.

See Also:

OWriteCache.internalFileId(long), OWriteCache.getId()

getMinimalNotFlushedSegment

public Long getMinimalNotFlushedSegment()

Specified by:

getMinimalNotFlushedSegment in interface OWriteCache

updateDirtyPagesTable

public void updateDirtyPagesTable(OCachePointer pointer,
                                  OLogSequenceNumber startLSN)

Specified by:

updateDirtyPagesTable in interface OWriteCache

create

public void create()

Specified by:

create in interface OWriteCache

open

public void open()

Specified by:

open in interface OWriteCache

addFile

public long addFile(String fileName,
                    long fileId)
             throws IOException

Specified by:

addFile in interface OWriteCache

Throws:

IOException

checkLowDiskSpace

public boolean checkLowDiskSpace()
                          throws IOException

Specified by:

checkLowDiskSpace in interface OWriteCache

Throws:

IOException

makeFuzzyCheckpoint

public void makeFuzzyCheckpoint(long segmentId,
                                byte[] lastMetadata)
                         throws IOException

Specified by:

makeFuzzyCheckpoint in interface OWriteCache

Throws:

IOException

flushTillSegment

public void flushTillSegment(long segmentId)

Specified by:

flushTillSegment in interface OWriteCache

exists

public boolean exists(String fileName)

Specified by:

exists in interface OWriteCache

exists

public boolean exists(long fileId)

Specified by:

exists in interface OWriteCache

restoreModeOn

public void restoreModeOn()
                   throws IOException

Specified by:

restoreModeOn in interface OWriteCache

Throws:

IOException

restoreModeOff

public void restoreModeOff()

Specified by:

restoreModeOff in interface OWriteCache

checkCacheOverflow

public void checkCacheOverflow()
                        throws InterruptedException

Specified by:

checkCacheOverflow in interface OWriteCache

Throws:

InterruptedException

store

public void store(long fileId,
                  long pageIndex,
                  OCachePointer dataPointer)

Specified by:

store in interface OWriteCache

files

public Map<String,Long> files()

Specified by:

files in interface OWriteCache

load

public OCachePointer load(long fileId,
                          long startPageIndex,
                          OModifiableBoolean cacheHit,
                          boolean verifyChecksums)
                   throws IOException

Specified by:

load in interface OWriteCache

Throws:

IOException

allocateNewPage

public int allocateNewPage(long fileId)
                    throws IOException

Specified by:

allocateNewPage in interface OWriteCache

Throws:

IOException

addOnlyWriters

public void addOnlyWriters(long fileId,
                           long pageIndex)

Specified by:

addOnlyWriters in interface OCachePointer.WritersListener

removeOnlyWriters

public void removeOnlyWriters(long fileId,
                              long pageIndex)

Specified by:

removeOnlyWriters in interface OCachePointer.WritersListener

flush

public void flush(long fileId)

Specified by:

flush in interface OWriteCache

flush

public void flush()

Specified by:

flush in interface OWriteCache

getFilledUpTo

public long getFilledUpTo(long fileId)

Specified by:

getFilledUpTo in interface OWriteCache

getExclusiveWriteCachePagesSize

public long getExclusiveWriteCachePagesSize()

Specified by:

getExclusiveWriteCachePagesSize in interface OWriteCache

deleteFile

public void deleteFile(long fileId)
                throws IOException

Specified by:

deleteFile in interface OWriteCache

Throws:

IOException

truncateFile

public void truncateFile(long fileId)
                  throws IOException

Specified by:

truncateFile in interface OWriteCache

Throws:

IOException

renameFile

public void renameFile(long fileId,
                       String newFileName)
                throws IOException

Specified by:

renameFile in interface OWriteCache

Throws:

IOException

close

public long[] close()
             throws IOException

Specified by:

close in interface OWriteCache

Throws:

IOException

close

public void close(long fileId,
                  boolean flush)

Specified by:

close in interface OWriteCache

restoreFileById

public String restoreFileById(long fileId)
                       throws IOException

Description copied from interface: OWriteCache

Finds if there was file in write cache with given id which is deleted right now. If such file exists it creates new file with the same name at it was in deleted file.

Specified by:

restoreFileById in interface OWriteCache

Parameters:

fileId - If of file which should be restored

Returns:

Name of restored file or null if such name does not exist

Throws:

IOException

checkStoredPages

public OPageDataVerificationError[] checkStoredPages(OCommandOutputListener commandOutputListener)

Specified by:

checkStoredPages in interface OWriteCache

delete

public long[] delete()
              throws IOException

Specified by:

delete in interface OWriteCache

Throws:

IOException

fileNameById

public String fileNameById(long fileId)

Specified by:

fileNameById in interface OWriteCache

nativeFileNameById

public String nativeFileNameById(long fileId)

Description copied from interface: OWriteCache

Obtains native file name by the given file id.

Native file name is a file name of a "physical" on-disk file, it may differ from the "virtual" logical file name.

Specified by:

nativeFileNameById in interface OWriteCache

Parameters:

fileId - the file id to obtain the native file name of.

Returns:

the obtained native file name or null if the passed file id doesn't correspond to any file.

getId

public int getId()

Specified by:

getId in interface OWriteCache

getCacheOverflowCount

public long getCacheOverflowCount()

getWriteCacheSize

public long getWriteCacheSize()

getExclusiveWriteCacheSize

public long getExclusiveWriteCacheSize()

setChecksumMode

public void setChecksumMode(OChecksumMode checksumMode)