Date: Thu, 28 Mar 2024 17:43:32 -0600 (MDT) Message-ID: <1271098496.61.1711669412515@localhost> Subject: Exported From Confluence MIME-Version: 1.0 Content-Type: multipart/related; boundary="----=_Part_60_1012452375.1711669412514" ------=_Part_60_1012452375.1711669412514 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Content-Location: file:///C:/exported.html
Object Stor= age provides the ability to t= urn any file and any associated metadata into an object accessible by an HT= TP API from anywhere. It ensures data integrity, provides data protection, = and scales to millions of objects without requiring traditional filesystem = management. Object Storage excels at handling unstructured data; and provid= es a location for you to store objects to be accessed by instances or if yo= u wish generally online as well.
Object Storage doesn=E2=80=99t have s= upport for a deep hierarchy; objects are stored in containers but container= s may not be inside another container. It=E2=80=99s possible to mimic folde= rs by using =E2=80=98/=E2=80=99 in an object=E2=80=99s name but from a syst= em point of view there is no folder support.
Our implementation of object storage = is powered by OpenStack=E2=80=99s Swift.
Static Web Assets
<= /li>Large amounts of unstructured= data (eg. map tiles, photos, etc.)
Archival (eg. tax records, ba= ckups, etc.)
Consistency/Protection=
Apps that leverage extensive = metadata on a file
You can gain access to your container= s and objects using several methods including the RAC Dashboard, the = swift command line client, interacting with the Swift API itself, or using = a popular GUI application. Of note at present we don=E2=80=99t offer the S3= compatibility API used by some libraries to connect to Swift. (If you woul= d like to use these feature please contact us at rac-admin@cybera.ca)
You can access your objects from anyw= here in the world - whether it's on one or more of your instances (eg. a sh= ared storage alternative), your personal computer, or elsewhere. By default= only you (when provided with your credentials) can access your objects, an= d with ACLs you can manage read and write access for other users on a conta= iner level.
We offer Swift in both the Calgary re= gion and the Edmonton region.
You can create, delete, copy, and dow= nload objects and containers using the RAC Dashboard. Additional features s= uch as setting metadata which is used for most advanced features are not av= ailable via the Dashboard.
If you haven=E2=80=99t already, it=E2=
=80=99s highly recommended you have reviewed Command-line To=
ols in the Advanced Guide. In=
stalling the python-openstackclient
should include installatio=
n of the python-swiftclient
, but if you need to manually insta=
ll it, follow the instructions below.
On Ubuntu:
pip ins= tall python-swiftclient
On MacOS, there may be a conflict wit= h the Apple Swift compiler, so be sure the Python bin dire= ctory is first in your $PATH (as specified in OSX command line tools):
pip ins= tall --user python-swiftclient
After sourcing your openrc file you c= an then use the swift command to interact with the object storage. Ex= amples can be found below.
Additionally your application can lev= erage the Swift API directly - either using curl or existing libraries (such as Boto f= or Python).
Our OpenStack Swift environment has s= upport for S3 API compatibility. This means that any application or library= which supports Amazon S3 should be able to be used in our Sw= ift environment.
To begin using the S3 API, you first = need to obtain a set of "EC2 Credentials". This is a set of credentials for= matted in the same way as Amazon's credentials. You can download your EC2 C= redentials by going here (https://cloud.cybera.ca/pro= ject/api_access/) and clicking "View Credentials".
Using a text editor of your choice, y= ou'll want to create a new file (eg. s3cred.sh) with the following details:=
export = AWS_ACCESS_KEY_ID=3Dfoo export AWS_SECRET_ACCESS_KEY=3Dbar
Alternatively you can find these credentials via the OpenStack CLI tools=
with the command openstack ec2 credentials list
.
The endpoint will be dependent on which region you are wishing to use - =
either yyc.cloud.cybera.ca:8080
or yeg.cloud.cyber=
a.ca:8080
For s3cmd the following example .cfg file can be used:
host_ba= se =3D https://swift-yyc.cloud.cybera.ca:8080 host_bucket =3D %(bucket)s.swift-yyc.cloud.cybera.ca:8080 access_key =3D <access key> secret_key =3D <secret key> use_https =3D True
To use the Edmonton region, change the yyc in the URLs to yeg.
Cyberduck is available for Mac OS X and Windows.
To use Cyberaduck using the standard = S3 compatibility, first obtain your EC2 Credentials, as described in the S3 API of this document.
Open Cyberduck and create a new Amazo= n S3 connection. Fill in the following information:
yyc.cloud.cybera.=
ca
(for the Calgary region) or yeg.cloud.cybera.ca
=
(for the Edmonton region).EC2_A=
CCESS_KEY
E=
C2_SECRET_KEY
Cyberduck provides built in support OpenStack Swift with Keystone V3. To = create a connection, create a new OpenStack Swift (Version 3) connection wi= th the details below:
More documentation is available on the Cyberduck Website.&n= bsp;
Server |
keystone-yyc.cloud.cybera.ca (Calgary) or keystone-= yeg.cloud.cybera.ca (Edmonton) |
Port |
5000 |
Username= span> |
project-na= me:Default:user-name (eg. joe@cybera.ca:Default:joe@cyb= era.ca - note the colons in between the project, domain name, and user = name) |
Password/S= ecret Key |
Your Passw= ord |
DO NOT upload files larger than 2 GB = using Cyberduck if you wish to share them via the Web.
See Swift CLI reference for more information.
swift s= tat -v swift stat -v container_name swift stat container_name file_name
swift p= ost container_name
echo 'H= ello World' > file_on_my_computer.txt swift upload container_name file_on_my_computer.txt
echo 'A= nother file' > another_file_on_my_computer.txt swift upload container_name another_file_on_my_computer.txt swift upload -m "X-Object-Meta-Hello: World" container_name another_file_on= _my_computer.txt swift stat -v container_name another_file_on_my_computer.txt
swift d= ownload container_name file_on_my_computer.txt
# Set a= container to be public swift post -r '.r:*' container_name # Allow another project to rw to container swift post -w 'second_project_id:*' container_name swift post -r 'second_project_id:*' container_name swift stat -v container_name
The second project may then interact =
with the shared container by appending the storage-url for the container to=
their swift commands:
swift s= tat -v --os-storage-url https://swift-<region>.cloud.cybera.ca:8080/v= 1/AUTH_<first_project_id> container_name
Set container to be public to anyone<= /span>
swift p= ost -r '.r:*' container_name
Figure out URL: by finding the AUTH_x=
xxx string by running swift=
stat
eg. http://swift.cloud.cybera.ca/v1/AUTH_xxxx/filename.txt or by using Cy= berduck (see above) and choose Copy URL.
Alternatively you can also enable web= listings so you can see all the objects in a container:
swift p= ost -r '.r:*,.rlistings' container_name swift post -m 'web-listings: true' container_name
And then visit http://swift.cloud.cybera.ca/v1/AUTH_xxxx/container_nam= e to view all the objects (changing = AUTH_xxxx for your AUTH information)
Another feature Swift offers is autom= atic versioning. This means every time an object is uploaded or updated it = will store the old version in a special versions container that is created.=
This means the older versions are sti= ll all accessible in a second container. You can enable this by setting the= =E2=80=98X-Versions-Location=E2=80=99 header on your container:
swift p= ost -m 'X-Versions-Location: myContainer-versions' myContainer
One last feature we wanted to highlig= ht is the ability to offer temporary URLs for objects or expiring URLs as t= hey are also called. This allows you to provide a URL that will stop functi= oning after a certain amount of time.
To do this you need to be able to set= a secret key in a header on your account to allow Temporary URLs, and then= create an HMAC-SHA1 signature generated from the HTTP method, the expirati= on timestamp, the path to the object, and the secret key set in the header = before. It=E2=80=99s not as complicated as it sounds however.
The first step is to create the secre= t key for your account:
swift p= ost -m 'X-Account-Meta-Temp-URL-Key: secretkeygoeshere'
Then we need to create our temporary = URL. You can use a tool such as swift-temp-url (Python), or have your applicati= on create the link by creating a signature based on the method, expiration = timestamp, and the path. The above linked tool uses the following code to d= o this:
import = hmac from hashlib import sha1 from time import time method =3D 'GET' expires =3D int(time() + 60) path =3D '/v1/AUTH_xxxx/container/object' key =3D 'mykey' hmac_body =3D '%s\n%s\n%s' % (method, expires, path) sig =3D hmac.new(key, hmac_body, sha1).hexdigest()
To use swift-temp-url - download the = file to your computer, and set the file to be executable. You can then run = it and get your URL. eg. a link that lasts for 10 minutes (600 seconds)&nbs= p;
echo ht= tps://swift-REGION.cloud.cybera.ca:8080$(swift-temp-url GET 600 /v1/AUTH_xx= xx/public_container/my_fancy_object secretkeygoeshere)
Results in a shareable l= ink:
Be sure to set set your = ACCOUNT ID, and then the path (proper container name and object name)
Swift segments files to support objec=
ts larger than 5 GB. If you want to upload a file larger than 5GB using the=
Swift CLI you will need to ensure you use the -S flag and set a segment si=
ze (below is 4GB segment size).
swift u= pload container_name -S 4294967296 my_large_fils.tar
By default Swift splits files larger than 5 GB while Cyberduck performs thi=
s for if the file is larger than 2GB. There are a couple caveats with=
the different approaches the two applications take to handling large files=
when it comes to accessing these files via a web browser (completely unrel=
ated to the size they use to split files).
Object uploaded via Swift CLI:=
Swift will create a <containername= _segments> container that holds the separate files. You=E2=80=99ll need = to enable read access AND rlistings for this container in order for the dow= nload to work. For a container named public you=E2=80=99ll need to make sur= e the following is set:
swift p= ost -r '.rlistings' public_segments
Once both are set, downloading the ob= ject via your web browser as an unauthenticated user will work as expected.= (Bug= #1082835)
Object uploaded via Cyberduck:=
Cyberduck creates a folder called .fi= le_segments in your container and places the segments in there. It then wri= tes JSON data to the manifest file so you can download the file later
Container sync offers the ability to = synchronize the contents of two or more containers between the two regions = of the Rapid Access Cloud or if supported, other OpenStack Swift installati= ons. Every five minutes, a sync will run copying any new or changed metadat= a and objects (including deletions) from the source container to the = destination container. Depending on the size of the objects it may take a c= ouple more minutes to copy the object in it's entirety.
This is done by creating a sync relat= ionship between any two swift containers. In the example below, we will cre= ate a container in Calgary (container1) and another in Edmonton (container2= ). Each container will sync to the other (two-way replication) which is use= ful if you want the objects to be highly available. We will need a key for = each container to share for security (the key will simply be =E2=80=98secre= t=E2=80=99, but for production please use a complex password).
Determine your AUTH ID (Ac= count ID, line 3):
$ sourc= e openrc-yyc $ swift stat -v StorageURL: https://swift-yyc.cloud.cybera.ca:8080/v1/AUTH_1a2b3c4d5e6f7890 Auth Token: this_is_not_a_real_token Account: AUTH_1a2b3c4d5e6f7890 Containers: 1 Objects: 116 Bytes: 103445941 Containers in policy "policy-0": 1 Objects in policy "policy-0": 116 Bytes in policy "policy-0": 103445941 X-Timestamp: 1418065112.84439 X-Trans-Id: txed09fb697477491b9af8d-0055e5fcce Content-Type: text/plain; charset=3Dutf-8 Accept-Ranges: bytes
Create or modify a container in C= algary and include the `--sync-to` and `-k` attributes (the `swift post` co= mmand updates info for a container, or creates one if it doesn=E2=80=99t ex= ist):
$ swift= post --sync-to '//rac/yeg/AUTH_1a2b3c4d5e6f7890/container2' -k 'secret' co= ntainer1
Create or modify a container in E= dmonton to sync to Calgary:
$ swift= post --sync-to '//rac/yyc/AUTH_1a2b3c4d5e6f7890/container1' -k 'secret' co= ntainer2
It is possible to do a one-way sync f= or disaster recovery, effectively creating a backup container; in our examp= le above, if we wanted to sync one-way from yyc =3D> yeg we create the c= ontainer in Calgary the same, but the container in Edmonton will not have a= `--sync-to` switch, only a `-k`:
$ swift= post -k 'secret' container2
Now objects in Calgary will replicate= to Edmonton, but not the other way.
At present disabling container sync i=
s not available in the openstack
command line tool at the=
time of writing. You must use the swift
command.
On= e example of why you might want to do this if you previously had a two-way container sync with YEG&nb= sp;=E2=86=94 YYC and only wish to= sync one-way now.
To remove the sync setting, provide a=
n empty --sync-to
parameter instead versus filling in the=
parameter to set up sync as seen in the section above.
$ swift= post --sync-to '' container
If you'd like to disable the sync entirely,= run the command on each container in both regions.