Problem solved 50% with #LeonidS help, Thanks!

Resolving Upload Delays Related to ZFS Sync Settings

When managing photo uploads, users may encounter delays or errors that can disrupt their experience. One common issue stems from the way the ZFS (Zettabyte File System) handles data synchronization, which can significantly impact upload performance. This article explores how adjusting the ZFS sync settings can resolve these upload-related errors.

Understanding the Problem

ZFS has a built-in synchronization mechanism that controls how data is written to disk. The default setting is sync=standard, which ensures data integrity by writing data to disk immediately. However, this can introduce latency, especially during bulk uploads, as each file must be fully written before the upload process can complete.

Symptoms of the Issue:

• Slow upload times when adding multiple photos to albums.
• Delayed responses in the application during the upload process.
• Possible notifications for each individual photo upload, causing notification overload.

Adjusting ZFS Sync Settings

To alleviate these issues, adjusting the ZFS sync setting for the relevant dataset can be beneficial. Setting sync=always ensures maximum data integrity but at the cost of performance. Conversely, setting it to sync=disabled allows for faster uploads, but it introduces a risk of data loss in the event of a crash.

For optimal performance in scenarios where data integrity is not as critical , you can set sync to disabled or standard based on your use case. But in this case it may cause an error (such as during photo uploads where users can re-upload failed photos due to lack of synchronization) so the best option is 'always'. Here's how to do this:

# Set sync to disabled for better performance during uploads
zfs set sync=always rpool/USERDATA/user_dataset

where 'user_dataset' is the folder that I want to make sure will be synchronized very quickly. You can have multiple datasets, For more information see the official documentation.https://docs.oracle.com/cd/E19253-01/819-5461/idx/index.html

There remains the problem of updating albums by adding new photos, which adds notifications for each photo.

To manage notifications effectively when adding new photos to albums, you can implement a centralized notification system that sends a single notification for the entire upload process rather than individual notifications for each photo. Here’s how you can approach this:

1. Batch Processing for Uploads

• Group Uploads: When a user uploads multiple photos at once, treat them as a single batch. You can modify the upload logic to queue the uploads and track the overall progress.

2. Notification Logic

• Single Notification: After all photos in the batch have been successfully uploaded, trigger a single notification indicating the total number of photos added.
• Failure Handling: If there are any failures during the upload, you can still notify the user about the successful uploads and inform them of any issues with specific photos if necessary.

3. User Interface Feedback

• Progress Indicator: Consider implementing a progress bar or indicator that shows the overall upload status. This gives users feedback during the upload process without flooding them with notifications.
• Completion Message: Once all uploads are complete, display a summary of the upload, including how many photos were successfully added.

4. Backend Adjustments

• If you’re using a backend service for uploads, ensure it supports batch processing and can handle the logic for grouping notifications before sending them to the frontend.

By centralizing notifications for album updates, you can enhance the user experience by reducing clutter and providing clearer feedback on the success of photo uploads. This approach can help users stay informed without being overwhelmed by multiple notifications for each individual photo.