Storage System

UNA's storage system provides a unified and abstracted interface for handling file uploads, management, storage, and delivery. Its core aim is to unify file storage, separating file management logic from the modules that use the files. This approach yields significant advantages:

  • Storage Flexibility: Files can be stored on the local server filesystem or seamlessly switched to remote cloud storage (like Amazon S3, Wasabi, etc.) via configuration, without requiring code changes in the modules using the files.
  • Organization: All managed files can be centralized (even on dedicated disks if needed), simplifying backups and infrastructure management.
  • Developer Simplicity: High-level PHP classes (BxDolStorage) handle complex operations like uploads, security token generation, and potentially image transformations, simplifying module development.
  • Quota Management: Administrators can set limits (size and number of files) per storage configuration to control resource usage.
  • Security: Provides mechanisms for private files, access control via expiring tokens, and centralized configuration of allowed file types/sizes.
  • Persistent Storage: Files uploaded via integrated components (like form uploaders) but not immediately associated with content (e.g., form not submitted) can persist temporarily ("orphans") and reappear upon page reload or later submission, improving user experience.

This document details the Storage system's concepts and how developers can interact with it, covering both the underlying structure and the recommended API usage.

Core Concepts

Storage Objects (sys_objects_storage)

A Storage Object is the central concept. It's a named configuration profile stored as a record in the sys_objects_storage table, defining the rules and backend for a specific category of files. Think of it as a virtual container definition within UNA.

Creating Storage Objects: While modules typically create these programmatically during installation (see "Defining Storage Objects" below), understanding the database table is key:

  • object (VARCHAR): Unique name for the storage object (e.g., bx_photos_processed, my_module_files). Convention: [vendor]_[module]_[description]. This name is used to get the storage instance in code.
  • engine (VARCHAR): Specifies the storage backend. Supported engines:
    • Local: Stores files on the server's local filesystem (default path: /storage/ in UNA root, configurable).
    • S3: Amazon S3 or S3-compatible storage (using v2 signature). Requires Access Key, Secret Key, Bucket, etc., configured in Studio Settings.
    • S3v4: Amazon S3 or S3-compatible storage (using v4 signature). Requires configuration like S3.
  • params (TEXT): PHP serialized string containing engine-specific parameters (often empty for Local).
  • token_life (INT): Lifetime (in seconds) for temporary security tokens used in URLs for private files.
  • cache_control (INT): Sets the Cache-Control: max-age= HTTP header value (in seconds) for served files. 0 disables explicit browser caching control.
  • levels (INT): Creates subdirectories based on the filename hash to distribute files and avoid limits on files per directory (e.g., level 2 for file.jpg might result in path ab/cd/remote_id.jpg). 0 disables this.
  • table_files (VARCHAR): Crucially, the name of the database table where metadata for files associated with this specific storage object is stored (see "Files Table" below).
  • ext_mode (ENUM allow-deny, deny-allow): File extension filtering mode:
    • allow-deny: Only allows extensions listed in ext_allow. ext_deny is ignored.
    • deny-allow: Allows all extensions except those listed in ext_deny. ext_allow is ignored.
  • ext_allow (VARCHAR): Comma-separated list of allowed extensions (e.g., jpg,png,gif,pdf). Used when ext_mode is allow-deny.
  • ext_deny (VARCHAR): Comma-separated list of denied extensions (e.g., exe,php,sh). Used when ext_mode is deny-allow.
  • quota_size (BIGINT): Maximum total size (in bytes) allowed for all files in this object. 0 for unlimited.
  • current_size (BIGINT): Automatically updated field showing the current total size of files.
  • quota_number (INT): Maximum number of files allowed in this object. 0 for unlimited.
  • current_number (INT): Automatically updated field showing the current number of files.
  • max_file_size (BIGINT): Maximum size (in bytes) allowed for a single uploaded file. Server limits (e.g., upload_max_filesize in php.ini) also apply if they are lower.
  • ts (INT): Unix timestamp of the last file upload operation for this object.

Configuration: Administrators fine-tune these settings (except object and table_files which are usually fixed by the module) via Studio > Storage.

Files Table (table_files)

Each Storage Object configuration points to a specific database table (table_files field) that stores the metadata for every file managed by that object. While you can define custom tables per object, the common practice, especially for modules using standard uploaders, is to use the default shared table: sys_storage_ghosts.

A typical files table structure (whether sys_storage_ghosts or custom) includes:

  • id (INT, PK, AI): Unique identifier for the file record. This is the ID developers usually store and work with.
  • profile_id (INT): The profile ID of the user who uploaded the file.
  • remote_id (VARCHAR, UNIQUE): A unique identifier for the file within its storage engine (often a hash or UUID, used in file paths/keys).
  • path (VARCHAR): The actual path or key used by the storage engine (relative for Local, key for S3).
  • file_name (VARCHAR): The original filename provided by the user during upload.
  • mime_type (VARCHAR): The detected MIME type of the file (e.g., image/jpeg, application/pdf).
  • ext (VARCHAR): The file extension (e.g., jpg, pdf).
  • size (INT): File size in bytes.
  • added (INT): Unix timestamp when the file was added/uploaded.
  • modified (INT): Unix timestamp when the file metadata was last modified.
  • private (INT): Flag indicating if the file is private (1) or public (0). Determines how URLs are generated and access is handled.

Developer Note: You typically only need to store the id from this table in your module's own tables to link your content to the uploaded file. You retrieve other details using the Storage API. Avoid direct manipulation of this table unless absolutely necessary.

Storage Engines

These are the backend implementations handling file operations:

  • Local: Stores files on the local server. Simple, default option.
  • S3 / S3v4: Stores files on Amazon S3 or compatible services. Offers scalability, durability, and potentially lower hosting costs for large amounts of data. Requires configuration of credentials and bucket details in Studio Settings.

Persistent Storage / Orphan Files

When using integrated uploaders (like the one in BxDolForm), files uploaded by a user might not be immediately associated with final content (e.g., the user uploads images but doesn't submit the form). These files are initially marked as "orphans." The Storage system keeps track of them. If the user returns to the form later, the uploader can display these previously uploaded orphan files.

  • Cleanup: Once a file ID is successfully saved and associated with content, developers must call $oStorage->afterUploadCleanup($iFileId, $iProfileId) to remove the file's orphan status. This prevents it from reappearing unnecessarily in future uploads on the same form/context.

Using Storage in Modules

Developers interact with the Storage system via the BxDolStorage class methods.

Getting a Storage Object Instance

Obtain an instance using the unique object name defined in sys_objects_storage (or preferably via your module's CNF configuration).

// Recommended: Get object name from module config
$sStorageObject = $this->_oConfig->CNF['OBJECT_STORAGE'];

// Or directly (less flexible):
// $sStorageObject = 'my_module_files';

$oStorage = BxDolStorage::getObjectInstance($sStorageObject);

if (!$oStorage) {
    // Handle error: Storage object configuration not found
    BxDolErrorHandler::getInstance()->log("Storage object not found: " . $sStorageObject);
    return false;
}
// Use $oStorage->... methods

Uploading Files

This is the standard and easiest way for user uploads within modules.

  1. Define a field with type: 'files' in your form array.
  2. Set the storage_object parameter to your object's name (use CNF).
  3. Configure other options like multiple, images_transcoder (for uploader previews), upload_buttons_titles.
  4. The form handles the upload process via its uploader widget. On successful submission, the submitted form data array will contain the file_id(s) under the field's name.
  5. Your code retrieves the file_id(s) from the submitted data and saves them.
  6. Important: After saving the IDs, call $oStorage->afterUploadCleanup($iFileId, $iProfileId) for each saved file_id to remove its orphan status.

(See previous documentation draft for a detailed BxDolForm field example)

Method 2: Programmatic Upload / Manual Form Handling

Use this when not using BxDolForm or when uploading programmatically.

  • storeFileFromForm($_FILES['input_name'], $bPrivate, $iProfileId, $iContentId = 0): Stores a file directly from the $_FILES superglobal array (from a standard HTML form upload).
    • $bPrivate: true or false to explicitly set file privacy (overrides object default if needed).
    • $iProfileId: ID of the uploader.
    • $iContentId: Optional ID to associate the file with content during upload (less common).
    • Returns the integer file_id on success, false on failure.
  • storeFileFromPath($sPath, $bPrivate, $iProfileId, $iContentId = 0): Stores from a local path.
  • storeFileFromString($sContent, $sFileName, $bPrivate, $iProfileId, $iContentId = 0): Stores from string data.
// Example using storeFileFromForm (like the one from the context)
require_once('./inc/header.inc.php'); // Standard UNA includes
// ... other includes ...
bx_import('BxDolStorage');

$sStorageObject = 'my_module_files'; // Use CNF constant in real module
$oStorage = BxDolStorage::getObjectInstance($sStorageObject);
$iProfileId = bx_get_logged_profile_id(); // Get current user

if ($oStorage && isset($_POST['add']) && isset($_FILES['file']) && $_FILES['file']['error'] == UPLOAD_ERR_OK) {
    $bPrivate = true; // Example: Store as private

    $mixedFileId = $oStorage->storeFileFromForm($_FILES['file'], $bPrivate, $iProfileId);

    if ($mixedFileId !== false) {
        $iFileId = (int)$mixedFileId;
        // --- IMPORTANT: Save the $iFileId ---
        // Your code here to associate $iFileId with your content in your module's DB table
        // saveMyModuleFileData($iContentId, $iFileId);

        // --- IMPORTANT: Cleanup Orphan Status ---
        $iOrphanCount = $oStorage->afterUploadCleanup($iFileId, $iProfileId);

        echo "Uploaded file ID: " . $iFileId . " (Cleaned up orphans: " . $iOrphanCount . ")";

    } else {
        // Handle upload error
        $sError = $oStorage->getErrorString(); // Get error message
        // Or: $iErrorCode = $oStorage->getErrorCode(); // Get numeric code
        echo "Error uploading file: " . $sError;
        BxDolErrorHandler::getInstance()->log("Storage error ($sStorageObject): " . $sError);
    }
}

// Basic HTML Form (for context)
/*
<form enctype="multipart/form-data" method="POST" action="store_file.php">
    Choose a file: <input name="file" type="file" /><br />
    <input type="submit" name="add" value="Upload" />
</form>
*/

Retrieving File URLs

  • getFileUrlById($iFileId): Gets the URL to access the file.
    • Public files: Returns a direct URL (local path or S3 URL).
    • Private files: Returns a temporary, tokenized URL pointing to storage.php (e.g., site.com/storage.php?o=...&f=...&token=...). Access is controlled by token_life and potentially module-level checks via alerts.
  • getImageUrlById($iFileId, $sTranscoderName = '', $bReturnOrigIfFailed = true): Gets URL for a specific transformed image version (defined in the Storage Object config). $sTranscoderName matches the transformation name (e.g., 'thumb', 'preview').
$oStorage = BxDolStorage::getObjectInstance($sStorageObject);
$iFileId = 1234; // The ID you saved earlier

if ($oStorage) {
    $sFileUrl = $oStorage->getFileUrlById($iFileId);
    $sImageUrl = $oStorage->getImageUrlById($iFileId); // Often gets original or a default preview
    $sThumbUrl = $oStorage->getImageUrlById($iFileId, 'thumb'); // Assuming 'thumb' transformation exists

    echo "File Link: <a href='{$sFileUrl}'>Download</a><br/>";
    if ($sImageUrl) {
        echo "Image: <img src='{$sImageUrl}' /><br/>";
    }
    if ($sThumbUrl) {
        echo "Thumbnail: <img src='{$sThumbUrl}' /><br/>";
    }
}

Retrieving File Information

  • getFileInfoById($iFileId): Returns an array with file metadata (name, size, mime_type, added timestamp, etc.) from the corresponding files table record.

Deleting Files

  • deleteFile($iFileId, $iProfileId = 0): Deletes the file record from the database table and the actual file from the storage engine (local disk or S3). Always check permissions before calling. Returns true on success.
// --- CRITICAL: Perform ACL permission check first ---
// if (!BxDolACL::getInstance()->isAllowed(...)) { return 'Access Denied'; }

$oStorage = BxDolStorage::getObjectInstance($sStorageObject);
if ($oStorage && $iFileId) {
    $bSuccess = $oStorage->deleteFile($iFileId);
    if ($bSuccess) {
        // Update your module's DB to remove the association
        // removeMyModuleFileData($iContentId, $iFileId);
    } else {
        // Log error
    }
}

Defining Storage Objects (Module Installation)

Modules should register their required Storage Objects during installation using the Installer class. This programmatically adds the record to sys_objects_storage and creates the necessary files table (if a custom one is specified, otherwise uses sys_storage_ghosts implicitly if table_files is empty).

// In BxMyModuleInstaller::install($aParams)

$this->addStorageObject([
    'object' => 'bx_my_module_files', // Use CNF constant name
    'engine' => 'Local', // Default engine
    'table_files' => '', // Use default sys_storage_ghosts
    'token_life' => 3600,
    'cache_control' => 2592000,
    'levels' => BX_DOL_STORAGE_LEVELS_DEFAULT, // Use system default levels
    'ext_mode' => 'allow-deny',
    'ext_allow' => ['jpg', 'jpeg', 'png', 'gif', 'pdf', 'zip'],
    'max_file_size' => 64 * 1024 * 1024, // 64MB
    // Quota fields usually left at 0 for module defaults (admin sets site-wide)
    'quota_size' => 0,
    'current_size' => 0,
    'quota_number' => 0,
    'current_number' => 0,
    'ts' => 0,
]);

// Don't forget removeStorageObject in the uninstall method!
// $this->removeStorageObject('bx_my_module_files');

(See previous documentation draft for image transformation definition example within addStorageObject)

Best Practices and Security Considerations

  • Use API: Always use BxDolStorage methods; avoid direct DB or filesystem access for managed files.
  • Permissions: Implement rigorous ACL checks before allowing file deletion or access to potentially sensitive private files.
  • Configuration: Use CNF constants for Storage Object names.
  • Store IDs: Only store the file_id in your module tables. Retrieve URLs and info dynamically.
  • Cleanup Orphans: Always call afterUploadCleanup() after successfully associating a file ID obtained via upload methods (especially storeFileFromForm or BxDolForm integration) with your content.
  • Uninstallation: Ensure your module's uninstall method calls removeStorageObject. Decide on a strategy for handling files associated with the module during uninstallation (delete them via API or leave them).
On This Page