Remote Storage

 I am committed long term with this project and know that my isp has file number limits that are not going to support a successful social media site. I have set up aws s3. I am ready to go public with the site and have a few files stored on local storage. I can sacrifice these if there is no way to upload to aws bucket but could be made public on individual file basis. Currently the bucket contains a storage folder with uploaded files may be 10-12 (hard to tell with so many dirs). Are files written to bucket root dir or can you use a folder in root dir? Since I set storage to aws i am getting file upload error :Oops! An error occurred with xxx.jpg file: error occurred. Please advise. By the way i appreciate all you help. Thanks again.
I see a discussion on Wasabi Cloud Storage with no end results Wasabi charges a fraction of Amazon's s3 and is s3 compliant. Is this a viable option? If so is there additional information on setup? Their site states :"This means any storage application that works with Amazon S3 will work with Wasabi (with no changes required to the storage application code)." The s3.php code needs to be modified. I counted 19 different references to amazon url.

  • 756
  • More
Replies (14)
    • Hello defendfreedoms !

      Yes, we have plans to expand the available external storage engines, but now earlier than in UNA 12.

      • Are files written to bucket root dir or can you use a folder in root dir?

        Files are written to bucket root dir, so in the bucket it should dirs like "bx_*"

        Since I set storage to aws i am getting file upload error :Oops! An error occurred with xxx.jpg file: error occurred. Please advise.

        We found some problem with European AWS S3 datacenter which are using some newer protocol which our library doesn't support, please try some other (particularly US) datacenter to see if this helps.

        I see a discussion on Wasabi Cloud Storage with no end results Wasabi charges a fraction of Amazon's s3 and is s3 compliant. Is this a viable option?

        You are correct, for now there are hardcoded calls to Amazon servers (in plugins/amazon-s3/S3.php file), so probably if you change these endpoints to Wasabi servers it could work, but I'm not sure, since I didn't try it yet.

        • I am in the process of changing all references from s3.amazonaws.com to wasabi counterpart urls on s3.php on test site. Will post results and make edited php available if successful.

          • have you been successful with this? I'm also looking for the same solution.

            • İ recommend objectivefs , with it you may mount almost any S3 system as a local drive and it works very fast. İt's a paid service but allows you to use it on 3 servers 

              Don't use riofs or similar they have dangerous bugs.

              • you can replace all the call from amazon to wasabi and still only have the data written to the bucket but not read. There  are other calls in the php script to amazon cloudfront I am not sure how to handle. Wasabi sees this issue strictly as the app developers responsibility.  Looking at their exponential growth I do not for-see any reason for them in the near future take on any developers responsibility. I requested information on the ease and procedures for transferring data from amazon to wasahi buckets incase una decides to have a scrlipt written for wasabi storage option.

                • As for amazon cloud storage I am using a US data center already. In the case of Wasabi I have changed all the bucket calls and able to write to the bucket but not read. There are other references to cloudfront in the script I am not sure how to handle. Maybe creating a community fund to have a script written for Wasabi might be a solution if enough interest is shown.
                  To have successful social media site, without filling it up with 3rd party advertising, is by memberships or a wealth philanthropist. Two major pillars a platform must have to accomplish this are; reliable up-to-date and affordable payment processors and offsite cloud storage options to sustain growth.

                  • what is the timeframe and with whom?

                    • As for amazon cloud storage I am using a US data center already. In the case of Wasabi I have changed all the bucket calls and able to write to the bucket but not read. There are other references to cloudfront in the script I am not sure how to handle. Maybe creating a community fund to have a script written for Wasabi might be a solution if enough interest is shown.
                      To have successful social media site, without filling it up with 3rd party advertising, is by memberships or a wealth philanthropist. Two major pillars a platform must have to accomplish this are; reliable up-to-date and affordable payment processors and offsite cloud storage options to sustain growth.

                      We're going to add more storages anyway including Wasabi - https://github.com/unaio/una/issues/1896 - but we can't tell when exactly.

                      • Still try to understand exactly what files are included in storage? I surmise it is videos, photos, and files that can be downloaded such as pdf text etc.. Any thing else?  Does all the folders and  files in the storage directory get put in s3 bucket? All i have looks like files created from cloudfront and no directories (there gone).

                        • Wasabi support informed me that there are 3rd party services that can transfer s3.bucket data from amazon to wasabi but it looks like you still have to use a service like cloudfront to deliver the file as a url from the bucket. From what I can understand they are primarly a cloud storage service (s3.bucket)

                          • This is the problem that I was facing. "you can replace all the call from amazon to wasabi and still only have the data written to the bucket but not read." Everything stores in Wasabi, but it shows a broken link in UNA. Also, for this, the github issue is still set to "future".

                            •  Alex T⚜️ I feel that we should solve the storage and bigint issues before we create enhancements and new features, because affordability and accessibility are the core of a successful social network. Without affordable storage along with, in this case, >2gb upload/encoding sizes, we can't unleash the full potential of the UNA software. Members of a social network are only willing to pay so much. Keeping the costs down for this site owner should be the number one priority.

                              •  Alex T⚜️ I feel that we should solve the storage and bigint issues before we create enhancements and new features, because affordability and accessibility are the core of a successful social network. Without affordable storage along with, in this case, >2gb upload/encoding sizes, we can't unleash the full potential of the UNA software. Members of a social network are only willing to pay so much. Keeping the costs down for this site owner should be the number one priority.

                                Support for other S3 compatible storages as well as Authentication Signature v4 have been added already and will be available in UNA v.12, we tested implementation using Amazon S3, Google Storage and Wasabi, documentation for this already added:
                                https://github.com/unaio/una/wiki/Remote-Storage

                                Login or Join to comment.