1. Home
  2. architecture
  3. storage

iconik Cloud Storage Architecture

iconik allows you to use your own cloud storage for storing original files, proxies or keyframes. To do this you configure a storage in iconik using credentials you generate using the storage vendor's administrative tools. This allows iconik to access the storage using these credentials and access information.

Terminology

This page uses the term pre-signed URLs throughout although this terminology only is used by Amazon S3 and Google Cloud Storage. The equivalent terms are Shared Access Signature for Azure Blob Storage and upload_url and Download URL for Backblaze B2.

Pre-signed URLs

The access credentials you configure in the storage settings are highly sensitive and usually gives complete control of all files in the storage. Because of this, access to these credentials are kept to a minimum. They are kept in encrypted storage and iconik only uses them internally.

Any external system which needs to access an object on a cloud storage has to request a URL using the APIs and will receive a pre-signed URL which can only be used to perform the specified operation and only on the object it requests access to. These pre-signed URLs are generated on demand but are cached for up to 24 hours for read-only operations, for performance reasons. The exact syntax for a pre-signed URL depends on the storage type but are generally

Presigned URLs are supported with all the storage vendors we support:

Search results

When searching in iconik's GUI, the search results page displays a preview image for each search result hit. This is called a keyframe and these are generated by the iconik transcoder on the initial ingest of a file into the system and are stored in a bucket of type Keyframe (either an iconik-managed bucket or a bring-your-own-bucket which is owned by the customer) The iconik search API responses optionally includes a link to a keyframe and a keyframe map representing a set of still frames of the asset. This is returned in the search result response to the client and client can then access the file via the object directly. In the iconik GUI this is used to display the search result pods. The client downloads the keyframe files directly from the cloud storage without going through the iconik SaaS service.

SearchingUserUsericonik Webiconik WebIconik APIIconik APICloud Storage BucketCloud Storage Bucket1Searches2Calls search API3Returns responseincluding pre-signedkeyframe URLsloop[for each search result]4Requests keyframe url(from the user's IP)5Returns file6Displays search result

Displaying a proxy

When displaying an asset page, iconik shows a proxy version of the asset. This can be either a video, an audio file or a preview image depending on the asset type. In order to get the proxy, the web GUI first makes a call to iconik's assets API to get general metadata about the asset and then uses that information to request a pre-signed URL for the proxy itself. It then creates a media element in the browser's DOM which makes the browser download and display the proxy to the user. The client downloads the proxy file directly from the cloud storage without going through the iconik SaaS service.

Viewing asset proxyUserUsericonik Webiconik WebIconik APIIconik APICloud Storage BucketCloud Storage Bucket1Opens assets page2Calls assets API3Returns asset metadata includingproxy file id4Calls files proxy API5Returns pre-signed URL for proxy file6Requests proxy url(from the user's IP)7Returns file8Displays assets page

Downloading a file

Downloading a file works very similarly to how a proxy is displayed. The client requests a pre-signed download URL for the file and then starts downloading it. The client may choose to download the file using concurrent connections using range requests if it wants to in order to increase the download speed.

DownloadingUserUsericonik Webiconik WebIconik APIIconik APICloud Storage BucketCloud Storage Bucket1Opens assets page2Calls assets API3Returns asset metadata includingfile id4Calls files API5Returns pre-signed URL for file6Requests file(from the user's IP)7Returns file8Displays downloaded filein Finder/Explorer

Uploading a file

Uploading a file works the same no matter which client is uploading. This includes the web browser GUI, the iconik Agent, the iconik Storage Gateway and any custom client.

In order to upload a file to iconik the client starts by creating all the required objects, including asset, format, file_set and file. This is explained in our API tutorial and will not be reiterated here. Once the file object has been created the client can request an upload URL. This is a pre-signed URL (or multiple URLs for some storage types) which can only be used to upload to a specific path on the cloud storage. Next, the client splits the file it wants to upload into chunks and starts uploading using as many concurrent connections as configured in the client settings. Once it has uploaded the entire file it issues a final call to iconik's API to mark the file as closed to signal that the upload has finished.

UploadingUserUsericonik Webiconik WebIconik APIIconik APICloud Storage BucketCloud Storage Bucket1Drags file into uploader2Calls API to createasset, format, file set and file3Returns ids for each object4Requests upload url for file id5Returns pre-signed URL for upload urlloop[for each file chunk]6Uploads chunk data to upload url(from the user's IP)7Returns success status8Calls files API to finalize upload9Returns success status10Displays assets page

Cloud Transcoder

The iconik cloud transcoder runs as a separate module in the iconik SaaS service. Even though it runs within the secure environment in iconik's backend it only uses pre-signed URLs in order to limit the risk of exposure to the storage credentials. When a transcode job is created it is queued internally in the iconik backend and when a free transcode slot is available the job is scheduled to a transcode node. The transcode process then fetches a pre-signed URL to download the source file, transcodes it to a temporary file. Once the transcode process is complete it uploads the resulting file to the proxy bucket and finally marks the proxy file as closed.

Cloud TranscoderUserUsericonik Webiconik Webiconik APIiconik APICloud TranscoderCloud TranscoderFiles APIFiles APISource BucketSource BucketProxy BucketProxy Bucket1Triggers transcode2Triggers transcode3Queues transcode job4Returns job id5Requests download URL for source file6Returns pre-signed URL for source fileloop[until transcode is done]7Requests file chunk8Returns file chunk9Transcodes to temp file10Request upload url for proxy11Returns pre-signed URL for proxy12Uploads proxy file13Returns success14Mark proxy file as closed15Return success

Edge Transcoder

The iconik Edge Transcoder is a variant of the iconik cloud transcoder which can be deployed to the customer's own cloud VMs. It can transcode files within the customer's VPC, without causing egress costs. The Edge Transcoder also uses pre-signed URLs in order to read data from the source file and to write data to the destination file.

Edge TranscoderUserUsericonik Webiconik WebEdge TranscoderEdge Transcodericonik APIiconik APISource BucketSource BucketProxy BucketProxy Bucket1Triggers transcode2Triggers transcode3Queues message4Calls transcode APIto get pending jobs5Returns transcode job6Requests download URL for source file7Returns pre-signed URL for source fileloop[until transcode is done]8Requests file chunk9Returns file chunk10Transcodes to temp file11Request upload url for proxy12Returns pre-signed URL for proxy13Uploads proxy file14Returns success15Mark proxy file as closed16Return success

Scanning a storage

In order to scan a cloud storage, iconik needs to communicate directly with the Storage API of the cloud storage service. To do this, iconik needs to use the access credentials directly. This cannot be done using pre-signed URLs in any of the cloud storage vendors we currently support. Because of this, scanning using the iconik Storage Gateway or Edge Transcoder is not possible. The scanner detects new file and creates assets, formats, filesets and files as needed. It also triggers transcodes if the scanner settings are configured to transcode on ingest.

Scanning storagesUserUsericonik APIiconik APIStorage ScannerStorage ScannerTranscode APITranscode APIFiles DatabaseFiles DatabaseCloud StorageCloud Storage1Scan the storage2Queue scan job3Return storagescan job id4Return storagescan job idloop[until all files have been listed recursively]5Retrieve list of files(from iconik's IP)6Returns list of filesloop[for each file]7Does file exist?8Responsealt[if file does not exist]9Create asset/format/fileset/file10Trigger transcode[if file exists]11update file data if needed

Transfering between storages

iconik gives users the ability to move or copy files between storages. Exactly how this works depends on the type and location of the source and destination storage. All transfers between storages are initiated from the iconik SaaS service. For a description of transfers to and from iconik Storage Gateways, please see the Hybrid Cloud documentation.

Buckets in the same storage account

Transfers between two storages within the same storage account in S3, GCS, Azure or B2 are performed using the storage vendor's API calls to trigger a copy without downloading the file. The exact mechanism for this differs between different storage vendors but it all follows the general schematics below.

Transfer within accountUserUsericonik APIiconik APITransfer HandlerTransfer HandlerFiles DatabaseFiles DatabaseCloud StorageCloud Storage1Initiate transfer2Queue transfer job3Return transfer job id4Return transfer job id5Create new file db entry6Start copying file to destinationloop[until transfer is finished]7Is transfer completed?8Return status9Mark file as closed

Between storage accounts or types

Transfers between storage accounts or storage types cannot be done server side in any of the storage types iconik supports. Instead, the iconik transfer system will download chunks of the file from the source storage and upload them to the destination storage. This means that there may be two egress costs, one for transferring the file from the source storage to iconik's SaaS service and one for transferring from iconik's SaaS service to the destination storage. The actual charges will depend on the policies of the various cloud service providers involved.

Transfer between accounts or typesUserUsericonik APIiconik APITransfer HandlerTransfer HandlerFiles DatabaseFiles DatabaseSource StorageSource StorageDestination StorageDestination Storage1Initiate transfer2Queue transfer job3Return transfer job id4Return transfer job id5Create new file db entry6Request pre-signed downloadURL for source file7Request pre-signed uploadURL for destination fileloop[until transfer is finished]8Is transfer completed?9Return file chunk10Upload chunk11Return response12Mark file as closed

Private VPC storage

Certain Cloud Storage providers allow a cloud storage to be configured to only be reachable from within the customer's own VPC. This is for example supported on Amazon S3, Google Cloud Storage and Azure Blob Storage. It is possible to use iconik in these scenarios but with some limitations and care has to be taken to configure the settings correctly. Any client which accesses the storage has to access it via an allowed address, for example using a VPN connection or using network peering. For information on exactly how to configure this, please contact your cloud storage vendor.

iconik's SaaS service will not be able to connect to the cloud bucket so scanning and transcoding using the iconik Cloud Transcoder will not be possible. In order to support transcoding you will either have to perform all transcoding on an iconik Storage Gateway, use an Edge Transcoder which runs on a VM inside the VPC or use a 3rd-party transcoding service which has access to the buckets.

Learn more