Storage

About Storage

Storage plans allow an account to control where data related to their account is stored. This can be critical when compliance with regulations is required.

A storage plan has three main components:

attachments: configuration for where to store attachments (binary data typically, like voicemails or faxes)
connections: connection information to various third-party storage sites
plan: a description of the storage plan(s) for the account.

Attachments

Rather than storing binary data like voicemails, received faxes, and call recordings, in the Piston databases, it can be convenient to store them in third-party storage services like Amazon S3, Google Drive, etc. This keeps the binary data in a place that an account or user maintains control over. Piston keeps a pointer to the location when it needs to fetch the binary data (such as when you call into your voicemail box).

The attachments object configures storage back-ends - for instance, if you want to store to S3, you'll add your AWS secret and key and your S3 bucket information here.

Connections

Connections can be used to point to an alternative CouchDB instance for storing the JSON documents or attachments (for instance: putting CDRs in their own cluster). These can be specified in the storage plans.

Plans

Plans determine what to do with certain classes of databases:

account: where to store account data
modb: where to store temporal data, like CDRs or voicemails
system: where to store system data, like prompts

Within the database classification, you can define things like the connection to use when reading/writing, what types of documents should be stored/retrieved, etc.

Enabling the storage endpoint

Crossbar must have the storage endpoint enabled first:

sup crossbar_maintenance start_module cb_storage

Schema

Key	Description	Type	Required
`^_`	Ignores CouchDB fields prefixed by underscores	`string() \| integer() \| boolean() \| object()`	`false`
`^pvt_`	Ignores Piston private fields prefixed by pvt_	`string() \| integer() \| boolean()`	`false`
`attachments`	Defines where and how to store attachments. Keys are 32-character identifiers to be used in storage plans	#/definitions/storage.attachments	`false`
`connections`	Describes alternative connections to use (such as alternative CouchDB instances	#/definitions/storage.connections	`false`
`id`	ID of the storage document	`string()`	`false`
`plan`	Describes how to store documents depending on the database or document type	#/definitions/storage.plan	`false`

storage.attachment.aws

schema for AWS attachment entry

Key	Description	Type	Required
`handler`	What AWS service to use	`string('s3')`	`true`
`settings.bucket`	Bucket name to store data to	`string(6..63)`	`true`
`settings.bucket_access_method`	how to access the host.	`string('auto' \| 'vhost' \| 'path')`	`false`
`settings.bucket_after_host`	use bucket after host as part of url	`boolean()`	`false`
`settings.host`	the s3 host, leave empty for default	`string(1..)`	`false`
`settings.key`	AWS Key to use	`string(1..128)`	`true`
`settings.port`	port to use	`integer()`	`false`
`settings.region`	the region where the bucket is located	`string(1..)`	`false`
`settings.scheme`	scheme to use to access host	`string('http' \| 'https')`	`false`
`settings.secret`	AWS Secret to use	`string(1..128)`	`true`
`settings`	AWS API settings	`object()`	`true`

storage.attachment.azure

schema for azure attachment entry

Key	Description	Type	Required
`handler`	What handler module to use	`string('azure')`	`true`
`settings.account`	the azure account name	`string()`	`true`
`settings.container`	the azure container where the files should be saved	`string()`	`true`
`settings.key`	the azure api key	`string()`	`true`
`settings`	Settings for the Azure account	`object()`	`true`

storage.attachment.dropbox

schema for dropbox attachment entry

Key	Description	Type	Required
`handler`	What handler module to use	`string('dropbox')`	`true`
`settings.oauth_doc_id`	Doc ID in the system 'auth' database	`string(1..)`	`true`
`settings`	Settings for the Dropbox account	`object()`	`true`

storage.attachment.google_drive

schema for google drive attachment entry

Key	Description	Type	Required
`handler`	What handler module to use	`string('google_drive')`	`true`
`settings.folder_id`	Folder ID in which to store the file, if any	`string()`	`false`
`settings.oauth_doc_id`	Doc ID in the system 'auth' database	`string(1..)`	`true`
`settings`	Settings for the Google Drive account	`object()`	`true`

storage.attachment.google_storage

schema for google storage attachment entry

Key	Description	Type	Default	Required	Support Level
`handler`	What handler module to use	`string('google_storage')`		`true`
`settings`	Settings for the Google Storage account	`object()`		`true`

storage.attachment.http

schema for HTTP(s) attachment entry

Key	Description	Type	Default	Required
`handler`	The handler interface to use	`string('http')`		`true`
`settings.base64_encode_data`	Toggles whether to base64-encode the attachment data	`boolean()`	`false`	`false`
`settings.send_multipart`	Toggle whether to send multipart payload when storing attachment - will include metadata JSON if true	`boolean()`		`false`
`settings.url`	The base HTTP(s) URL to use when creating the request	`string()`		`true`
`settings.verb`	The HTTP verb to use when sending the data	`string('post' \| 'put')`	`put`	`false`
`settings`	HTTP server settings	`object()`		`true`

storage.attachment.move_it

schema for MoveIt attachment entry

Key	Description	Type	Required
`handler`	What MoveIt service to use	`string('move_it')`	`true`
`settings.folder_id`	Folder ID to store data to	`string()`	`true`
`settings.host`	the MoveIt host, including the protocol and port	`string(1..)`	`true`
`settings.password`	MoveIt password to use	`string(1..128)`	`true`
`settings.username`	MoveIt username to use	`string(1..128)`	`true`
`settings`	MoveIt API attachment storage settings	`object()`	`true`

storage.attachment.onedrive

schema for OneDrive attachment entry

Key	Description	Type	Required
`handler`	What handler module to use	`string('onedrive')`	`true`
`settings.oauth_doc_id`	Doc ID in the system 'auth' database	`string(1..)`	`true`
`settings`	Settings for the OneDrive account	`object()`	`true`

storage.attachments

Defines where and how to store attachments. Keys are 32-character identifiers to be used in storage plans

Key	Description	Type	Required
`^[a-z0-9]{32}$.name`	Friendly name for this configuration	`string()`	`false`
`^[a-z0-9]{32}$.settings.field_list`	list of fields to compose destination url	`["array(", "[#/definitions/storage.attachments.field](#storageattachments.field)", ")"]`	`false`
`^[a-z0-9]{32}$.settings.field_separator`	toplevel, field separator to compose destination url	`string()`	`false`
`^[a-z0-9]{32}$.settings.folder_base_path`	base folder path	`string()`	`false`
`^[a-z0-9]{32}$.settings`	Settings all handlers implement	`object()`	`false`
`^[a-z0-9]{32}$`	Configuration for the supported storage backends	`object()`	`false`

storage.attachments.field

field used when composing destination url

Key	Description	Type	Default	Required	Support Level

storage.connection.couchdb

schema for CouchDB connection entry

Key	Type	Required
`driver`	`string('Piston_couch')`	`true`
`name`	`string()`	`false`
`settings.connect_options.keepalive`	`boolean()`	`false`
`settings.connect_options`	`object()`	`false`
`settings.connect_timeout`	`integer()`	`false`
`settings.credentials.password`	`integer()`	`true`
`settings.credentials.username`	`string()`	`true`
`settings.credentials`	`object()`	`false`
`settings.ip`	`string()`	`true`
`settings.max_pipeline_size`	`integer()`	`false`
`settings.max_sessions`	`integer()`	`false`
`settings.pool.name`	`string()`	`true`
`settings.pool.size`	`integer()`	`true`
`settings.pool`	`object()`	`false`
`settings.port`	`integer()`	`true`
`settings`	`object()`	`true`

storage.connection.couchdb_and_postgresql

schema for couchdb and postgresql connection entry

Key	Type	Required
`driver`	`string('Piston_couchdb_and_postgresql')`	`true`
`name`	`string()`	`false`
`settings`	`object()`	`true`

storage.connection.postgresql

schema for postgresql connection entry

Key	Type	Required
`driver`	`string('Piston_postgresql')`	`true`
`name`	`string()`	`false`
`settings.connect_timeout`	`integer()`	`false`
`settings.connection_pool_max_overflow`	`integer()`	`false`
`settings.connection_pool_size`	`integer()`	`false`
`settings.credentials.password`	`string()`	`true`
`settings.credentials.username`	`string()`	`true`
`settings.credentials`	`object()`	`false`
`settings.ip`	`string()`	`true`
`settings.port`	`integer()`	`true`
`settings`	`object()`	`true`

storage.connections

Describes alternative connections to use (such as alternative CouchDB instances

Key	Description	Type	Default	Required	Support Level
`^([a-z,0-9]){32}$`		#/definitions/storage.connection.couchdb	#/definitions/storage.connection.postgresql	#/definitions/storage.connection.couchdb_and_postgresql
`local`		`object()`		`false`

storage.plan

Describes how to store documents depending on the database or document type

Key	Description	Type	Required
`account`	schema for database storage plan	#/definitions/storage.plan.database	`false`
`modb`	schema for database storage plan	#/definitions/storage.plan.database	`false`
`system`	schema for database storage plan	#/definitions/storage.plan.database	`false`

storage.plan.database

schema for database storage plan

Key	Description	Type	Required
`attachments`	schema for attachment ref type storage plan	#/definitions/storage.plan.database.attachment	`false`
`connection`		`string()`	`false`
`database.create_options`		`object()`	`false`
`database`		`object()`	`false`
`types.call_recording`	schema for document type storage plan	#/definitions/storage.plan.database.document	`false`
`types.cdr`	schema for document type storage plan	#/definitions/storage.plan.database.document	`false`
`types.fax`	schema for document type storage plan	#/definitions/storage.plan.database.document	`false`
`types.mailbox_message`	schema for document type storage plan	#/definitions/storage.plan.database.document	`false`
`types.media`	schema for document type storage plan	#/definitions/storage.plan.database.document	`false`
`types`		`object()`	`false`

storage.plan.database.attachment

schema for attachment ref type storage plan

Key	Description	Type	Required
`handler`		`string()`	`false`
`params.folder_path`	folder path	`string()`	`false`
`params`		`object()`	`false`
`stub`		`boolean()`	`false`

storage.plan.database.document

schema for document type storage plan

Key	Description	Type	Default	Required	Support Level
`attachments`	schema for attachment ref type storage plan	#/definitions/storage.plan.database.attachment		`false`
`connection`		`string()`		`false`

Fetch

GET /v2/accounts/{ACCOUNT_ID}/storage

curl -v -X GET \
    -H "X-Auth-Token: {AUTH_TOKEN}" \
    http://{SERVER}:8000/v2/accounts/{ACCOUNT_ID}/storage

{
    "auth_token": "{AUTH_TOKEN}",
    "data": {
        "attachments": {
            "{UUID}": {
                "handler": "s3",
                "name": "S3 Storage",
                "settings": {
                    "bucket": "{S3_BUCKET}",
                    "key": "{AWS_ACCESS_KEY}",
                    "secret": "{AWS_SECRET_KEY}"
                }
            }
        },
        "id": "{ACCOUNT_ID}",
        "plan": {
            "modb": {
                "types": {
                    "mailbox_message": {
                        "attachments": {
                            "handler": "{UUID}"
                        }
                    }
                }
            }
        }
    },
    "request_id": "{REQUEST_ID}",
    "revision": "{REVISION}",
    "status": "success"
}

Create

PUT /v2/accounts/{ACCOUNT_ID}/storage

curl -v -X PUT \
    -H "X-Auth-Token: {AUTH_TOKEN}" \
    http://{SERVER}:8000/v2/accounts/{ACCOUNT_ID}/storage

For instance, setting up your HTTP server to receive new voicemails for the account:

{
  "data": {
    "attachments": {
      "{UUID}": {
        "handler": "http",
        "name": "My HTTP server",
        "settings": {
          "url": "http://my.http.server:37635/some_prefix",
          "verb": "post"
        }
      }
    },
    "plan": {
      "modb": {
        "types": {
          "mailbox_message": {
            "attachments": {
              "handler": "{UUID}"
            }
          }
        }
      }
    }
  }
}

Change

POST /v2/accounts/{ACCOUNT_ID}/storage

curl -v -X POST \
    -H "X-Auth-Token: {AUTH_TOKEN}" \
    http://{SERVER}:8000/v2/accounts/{ACCOUNT_ID}/storage

Patch

PATCH /v2/accounts/{ACCOUNT_ID}/storage

curl -v -X PATCH \
    -H "X-Auth-Token: {AUTH_TOKEN}" \
    http://{SERVER}:8000/v2/accounts/{ACCOUNT_ID}/storage

Remove

DELETE /v2/accounts/{ACCOUNT_ID}/storage

curl -v -X DELETE \
    -H "X-Auth-Token: {AUTH_TOKEN}" \
    http://{SERVER}:8000/v2/accounts/{ACCOUNT_ID}/storage

Fetch

GET /v2/accounts/{ACCOUNT_ID}/storage/plans

curl -v -X GET \
    -H "X-Auth-Token: {AUTH_TOKEN}" \
    http://{SERVER}:8000/v2/accounts/{ACCOUNT_ID}/storage/plans

Create

PUT /v2/accounts/{ACCOUNT_ID}/storage/plans

curl -v -X PUT \
    -H "X-Auth-Token: {AUTH_TOKEN}" \
    http://{SERVER}:8000/v2/accounts/{ACCOUNT_ID}/storage/plans

Fetch

GET /v2/accounts/{ACCOUNT_ID}/storage/plans/{STORAGE_PLAN_ID}

curl -v -X GET \
    -H "X-Auth-Token: {AUTH_TOKEN}" \
    http://{SERVER}:8000/v2/accounts/{ACCOUNT_ID}/storage/plans/{STORAGE_PLAN_ID}

Change

POST /v2/accounts/{ACCOUNT_ID}/storage/plans/{STORAGE_PLAN_ID}

curl -v -X POST \
    -H "X-Auth-Token: {AUTH_TOKEN}" \
    http://{SERVER}:8000/v2/accounts/{ACCOUNT_ID}/storage/plans/{STORAGE_PLAN_ID}

Patch

PATCH /v2/accounts/{ACCOUNT_ID}/storage/plans/{STORAGE_PLAN_ID}

curl -v -X PATCH \
    -H "X-Auth-Token: {AUTH_TOKEN}" \
    http://{SERVER}:8000/v2/accounts/{ACCOUNT_ID}/storage/plans/{STORAGE_PLAN_ID}

Remove

DELETE /v2/accounts/{ACCOUNT_ID}/storage/plans/{STORAGE_PLAN_ID}

curl -v -X DELETE \
    -H "X-Auth-Token: {AUTH_TOKEN}" \
    http://{SERVER}:8000/v2/accounts/{ACCOUNT_ID}/storage/plans/{STORAGE_PLAN_ID}

Skipping attachment settings validation

When a storage plan is PUT/POSTed to Crossbar, Piston will attempt to use the attachments' settings and store a small text file to verify that files can be stored remotely. Piston will then issue a GET request to read the file back to test retrieval.

For "dumb" storage backends this is typically a non-issue as storing/retrieving files is what the backend does!

For "smart" backends, where a custom handler (like an HTTP web app) is accepting the files, adding code to handle this test file's storage/retrieval could place an unnecessary burden on the backend or be redundant after the first test if using the same destination for all accounts. As such, a request parameter can be included to skip this portion of the validation:

curl -v -X PUT \
    -H "X-Auth-Token: {AUTH_TOKEN}" \
    -H "Content-Type: application/json" \
    -d '{"data":{...}}' \
  http://{SERVER}:8000/v2/accounts/{ACCOUNT_ID}/storage?validate_settings=false

!!! danger If the storage backend is unable to process the storage request, you could lose the data attempting to be stored.

Enabling This Feature

By default, Piston will not allow clients to skip settings validation. Clients that include the validate_settings request parameter on these systems will receive a 400 validation error indicating attachment storage settings must be tested.

Sysadmins can allow clients by setting a system_config flag: sup kzs_plan allow_validation_overrides

Disabling it later is similar: sup kzs_plan disallow_validation_overrides

URL formatting

It is possible to craft the URLs used by the handler based on the JSON document and attachment being saved by specifying a field_list array of objects that will help Piston map values to the generated URL:

{UUID}:{
    "handler":"{HANDLER}",
    "settings":{
        "field_list":[
            {"arg":"account_id"}
            ,{"arg":"id"}
            ,{"arg":"attachment"}
        ],
        "url":"http://base.your.domain/"
    }
}

In this case (the default for the HTTP handler) the URL provided in the handler's settings will be appended with /{ACCOUNT_ID}/{DOC_ID}/{ATTACHMENT_NAME}.

Field Options

list of fields to compose destination url

Key	Description	Type	Required
`arg`	a argument passed to the handler	`string('account_id' \| 'db' \| 'id' \| 'attachment')`	`false`
`const`	a constant value added to the string	`string()`	`false`
`field`	a field from the metadata document	`string()`	`false`
`group`	group the inner fields definitions with an empty separator	`array()`	`false`

Examples

Given a base URL of http://my_server.com/storage, an attachment call.mp3 being stored in the account000 account on the abc123 doc defined below:

{"_id":"abc123"
 ,"foo":"bar"
 ,"bing":"bang"
}

We can create the following generated URLs:

| URL | field_list | | | http://my_server.com/storage/account000/abc123/call.mp3 | [{"arg":"account_id"}, {"arg":"id"}, {"arg":"attachment"}] | | | http://my_server.com/storage?path=/account000/abc123/call.mp3 | [{"const":"?path="}, {"arg":"account_id"}, {"arg":"id"}, {"arg":"attachment"}] | | | http://my_server.com/storage/bar/call.mp3 | [{"field":"foo"}, {"arg":"attachment"}] | | | http://my_server.com/storage/account001_call.mp3 | [{"group":[{"arg":"account_id"}, {"const":"_"}, {"arg":"attachment"}]}] | |

Global Storage settings

If you are the super-duper admin you can setup storage policy across your cluster by using the URL /v2/storage instead of per-account.

The validation request will arrive with /system_data/{DOC_ID}/{ANAME} on the end of the URL.

If you need to revert back to the standard settings you can run sup kzs_plan reset_system_dataplan; and changes will be removed and the initial version reinstalled.

Keys	Action
`?`	Open this help
`n`	Next page
`p`	Previous page
`s`	Search