 |
|||||||||||||||||||||||||||||||
|
|||||||||||||||||||||||||||||||
The storage layer is responsible for loading a source media and writing a result file for every transcoding job. Alembik media processors are supposed to access original media files and save the transcoded results through that layer. Such approach enables a significant performance improvement whenever there are more transcoding requests for the same source file (and with the same set of transcoding parameters set), particularly if the media is located at a remote site. Currently Alembik provides a powerful storage system using the local file system; the actual storage directory is determined by Configurator module. The evaluation of distinct filenames is based on a hashing algorithm based on the original file location and the parameter values of the transcoding request (more details can be found in the stored files naming section). That approach lets the server identify unambiguously medias in Alembik storage and check whether the required source or transcoding output file does not already exist. When asked for an access to a source file the LocalStorage component evaluates its own local filename and if there is no file present under the evaluated location it delegates loading of a requested file to a chosen implementation of the MediaLoader interface. The picture below presents the flow of control, though in reality a source file is also requested during the request preprocessing.
The storage architecture introduces the (org.alembik.storage.Storage) interface, which could be implemented to provide a custom storage module. A Storage instance is always retrieved through the org.alembik.storage.StorageFactory class, which returns a preconfigured implementation. At the moment the only implementation provided and returned is org.alembik.storage.local.LocalStorage.
  As mentioned above the implementations of the MediaLoader interface are used by LocalStorage to obtain a source file copy in the local file system. At the time of writing there are only two implementations available: FileMediaLoader and HttpMediaLoader, which are used to load files from the local file system and from the net (HTTP protocol) respectively. It is the client's responsibility to set the proper media loading protocol for accessing any source file. The server always looks for the SourceProtocol property in the TranscodingJob.Source's extension data (accepted values are now: HTTP and FILE) and if there is none set it assumes the HTTP loading mode (for more details see TranscodingUtils.SourceProtocol class). In case of FILE protocol and the progressive segmentation in HTTP one, a client may also opt for not copying an original source file into the Alembik storage. Then the noCopySource property set to true must be placed in TranscodingJob.Source's extension data; see also the two TranscodingUtils.setSourceProtocol() methods provided for user convenience.
The first method: load() is responsible for fetching and putting a source file for a given TranscodingJob into an appropriate storage directory (see below). The modeSecure set to true tells MediaLoader that there might be other parallel process loading the same file. The lastModified() method returns the modification timestamp of the original remote file (by which the storage might decide whether the file re-loading is necessary), while evaluateContentType() provides the remote file's mime-type.
Hereafter we explain the method currently used by LocalStorage to compute the name for source and result files. Both evaluations are based on the Java hashing algorithm implemented in org.alembik.storage.NamingUtils class. As previously stated the storage reads and writes all source and result files from/to a directory, which is determined in the Configurator module. There it maintains a structured tree of folders. Every leaf of this tree (i.e. folder) may contain a single source media file and all its transcoded results. During each transcoding operation the storage needs to compute a proper folder, where source file and its results are to be stored. Folder path is determined by the original location of the source file provided by TranscodingJob.Source object). In the case of a remote media file (HTTP loading protocol) the path consists of three folders. First goes the hashcode of a host name modulo 10000, then the hashcode of a full original location modulo 10000 and finally the very same hashcode unaltered. In case there is a DownloadLimit transformation applied, the last element is suffixed with the relevant number of bytes (size limit) or seconds (approximate duration).
Note, that if choosing any "progressive segmentation" option, the suffix of subsequent folders will contain the relevant size/duration range, e.g. '-200-400K' or '-20-30S', where 'K' stands for KBs and 'S' for seconds. The three levels reduce the risk of producing too many folders within a single directory, whose number is usually restricted by operating systems. The modulo operation keeps the number of folders lesser than 20000 entries on the first two levels. In the case of a local media files the risk of reaching excessive number of folder is significantly lower. Hence there is created only two folders structure: the first level with the hashcode of the media location modulo 10000 and the second one with the same hashode left intact. For example:
The source file is then stored in the computed directory with the name src followed by the extension corresponding to its mime-type (in the example case "src.jpg"). The last date modification of the original file is saved in a special marker file, which gives the storage an opportunity to reload the source file, whenever it gets changed at its remote location. The marker file has the same name as the source file followed from the extension ".ok" (in this case: "src.jpg.ok"). A bit more complicated is the evaluation of the result file name. The storage prepares an ordered collection of all transcoding parameters of the processed job and forms them into a structured string. Below there is the example of a job, which requests an original image to be resized and rotated for 90 degrees as well as its mimetype be changed from 'jpg' to 'gif'. In this case the system will create the following list of parameters:
One may observe that in the list of the attributes used to create a filename, the name of the source media is not present; it is because the source location is already involved in the directory path evaluation; hence all the files in a leaf directory are "trasformation results" of the same source file, which is also placed there. The are two main exceptions for the aforementioned file management. The first occurs in the situation, when client specifies properties of ExternalLocation object within TranscodingJob.Target instance. The path parameter overrides the Storage-evaluated folders path for a transcoded file, while the additional name parameter forces the actual file name (excluding its extension). The second exception comes with the optional internal path prefix feature. Alembik client may request inserting an additional folder to the tree structure at the root level. If so, the source and/or transcoded folders (with all their structure and naming preserved) will be placed in a specified folder inside the storage root directory. The "prefix" folder can be declared by setting the appropriate property in TranscodingJob.Target's extension data: sourcePathPrefix for a source file and targetPathPrefix for a transcoded one. There are three utility methods made available in TranscodingUtils class: setInternalSourcePathPrefix(), setInternalTargetPathPrefix() and the combination of both: setInternalPathPrefix().
The algorithm used in Alembik to determine the mime-type of a specific media is implemented to the class org.alembik.ContentTypeDetector. The mime-type detection process always takes place when user leaves the Source's contentType field empty. It also depends on the media loading protocol chosen for the transcoding job. In case of remote medias (loaded with the HTTP protocol) the detector checks first the content-type header of the HTTP response. Later on, when a media file gets downloaded, it gets analyzed by the MimeUtil tool, which evaluates the mime-type on the basis of first bytes (magic numbers) of the file. Then the detector compares two values; if they are equal it assumes the result. Otherwise it obtains the third value from an internal mime-types repository (see org.alembik.util.MimeTypesRepository), which resolves mime-types against file extensions. If it coincides with one of the previous values, it is assumed to be correct; if not, the HTTP header-based value takes over. In case of local medias (when the FILE loading protocol is used) the mime-type detection process is simplier; the analysis is started with the aforementioned MimeUtil tool. Then if it cannot evaluate any mime-type the detector returns the result of resolution performed by MimeTypesRepository described above.
In Alembik the reload policy is a notion that determines, when a resource ought to be reloaded. The assumed policy depends on the presence of SourceReloadPolicy property in the TranscodingJob.Source's extension data. There are currently four values supported: ALWAYS, IF_OLDER_THAN, ONLY_ONCE and NEVER (see TranscodingUtils.setReloadPolicy() method). In general Alembik has to load a resource first, before starting all transcoding proccess on that file. It fetches the file only if it is missing (not loaded yet) or if it gets outdated (its remote version has changed). This default reload policy is called IF_OLDER_THAN with the number of "no-checking" minutes set to 15 (the server will not touch the file for the next 15 minutes after its downloading). To check whether a stored file is outdated Alembik compares its original modification date (saved into the marker file at the previous loading time) with the one of the remote file, provided by MediaLoader's lastModified() method. A client may change the number of "no-checking" minutes for the default strategy, which will force Alembik to keep the local file unchanged (no matter if it is up-to-date or not) for that specified amount of time (starting from the completion of its download). Such IF_OLDER_THAN's "no-checking" period can be specified via TranscodingUtils.setIfOlderThanTime() method, which puts SourceOlderThan property into the TranscodingJob.Source's extension data. The second policy is called ALWAYS and it simply forces to Alembik to reload a source file. Finally, the client may decide that a file should not be reloaded by any means once it is in the storage (ONLY_ONCE) or that it should be never loaded, which assumes that it is already there (NEVER).
Generally Alembik anticipates raw (unencoded and unescaped) URLs passed via the TranscodingJob.Source's location property, which are processed internally before requesting any resource. However, if a remote server requires a specific URL format the client may request Alembik to give up any location processing and use the URL as it is. Then srcLocEscaped property has to set to true inside the TranscodingJob.Source's extension data (see org.alembik.util.TranscodingUtils.setSourceLocationEscaped() method).
 Certain servers (e.g. youtube.com) make their media content available through temporary random-generated URLs. In that case using them in a regular way would lead to frequent downloading of the same media file. To avoid it there has been introduced a new identityUrl property to be passed within the TranscodingJob.Source's extension data. Thus whenever a client specifies that URL (e.g. via org.alembik.util.TranscodingUtils.setIdentityUrl() method), it is used for the local filename evaluation instead of the regular one (i.e. the source location). Then the media files obtained through temporal links will not be reloaded needlessly, providing that one may identify each file unanimously with some identity URL.
 In Alembik we may define two processing blocks (here called tasks) that are executed whenever a transcoding request arrives. The first is media loading task, which stands for the process of getting a source file, and the other is transcoding task, which carries out the source file transcoding. These tasks must be synchronized in order to obtain a real thread-safe execution flow. The synchronization of the tasks has several requirements imposed:
The basic synchronization utility in Alembik is org.alembik.synchronizer.SynchronizerImpl that implements the org.alembik.synchronizer.Synchronizer interface.
This interface defines four methods, two for blocking and unblocking a file, and other two for blocking and unblocking a critical section. The file blocking methods ensure that no concurrent threads can access a given file in parallel. For example they are used to synchronize loading of the same source file or transcoding into the same result file. The critical sections methods play a different role. They are designed to synchronize processing in parallel, for example reloading a source file with several transcoding processes already using it. The startCriticalSection() method opens a critical section, but its behaviour depends on the critical section's operation (UPDATING or READING). For one particular file only one UPDATING section can be opened, though several threads may start its READING sections in parallel. The UPDATING and READING sections are mutually exclusive: the task of one type has to wait until all tasks of the other type are completed (and while waiting it prevents from starting any new task of a different type).
|
|||||||||||||||||||||||||||||||
STORAGE
