IMPORTANT

• An issue was found in the migration below that can affect application start commands. This release should not be used, a fix is being worked on for CAPI 1.7.0.

CC API Version: 2.63.0

Service Broker API Version: 2.10

CCDB Migration for V3 API

This release includes a significant migration of the CCDB that is the first step to releasing the CC V3 API.

ALL EXPERIMENTAL V3 DATA WILL BE REMOVED WITH THIS MIGRATION

The migration will cause temporary API downtime as Cloud Controller jobs that have not been updated will not be able to use the migrated database schema. Users will not be able to use the CC API to push apps, update apps or get app information, but it will not affect apps currently deployed to Cloud Foundry.

User Impact

• Apps will continue to run normally.
• The Cloud Controller API will be unavailable or raise errors during the deploy meaning users will be unable to push apps, update existing apps or get app information.
• Crashing apps will not be restarted while the Cloud Controllers are being deployed.
• App logs will not be aggregated to syslog drains while the Cloud Controllers are being deployed if the deployment takes longer than the “properties.syslog_drain_binder.drain_url_ttl_seconds” manifest property. The default value for this property is 60 seconds.

Operator Procedure

Operators are not required to do anything differently for this deploy, but due to the size of the migration, operators may wish to take some additional steps to minimize user facing errors. See the Deployment Options below for information on steps an operator may take for this deployment.

1. Ensure there is a current CCDB backup prior to deploying.
2. Ensure there is adequate disk space available on the database server. This migration moves around significant amounts of data in the CCDB. The amount of disk space required will vary by RDBMS, it is recommended to err on the safe side.
3. Alert consumers of the API that there is a maintenance period, specifically any GUI client may want to display a maintenance page.
4. Consider temporarily increasing the value of “properties.syslog_drain_binder.drain_url_ttl_seconds” to prevent app logs failing to propagate to syslog drains during the deploy.
5. Select a deployment method from below and deploy.

Deployment Options

Deploy Normally

This is the simplest option. It should result in the shortest downtime period, but may be the most surprising to users as requests will appear to fail randomly as a result of a rolling deploy.

Procedure

1. Deploy the updated cf-release

Impact

• Users and clients will receive 500 errors from the API.
• Cloud Foundry components continue to attempt to access the database resulting in errors until the deploy is completed.

Scale API jobs to zero instances

This option prevents access to CCDB by Cloud Foundry components that do not know the updated schema. It also prevents clients from seeing seemingly random errors; they will only see unavailability errors.

Procedure

1. Prior to deploying the new cf-release, perform a manifest only deploy that scales down the following jobs
   • api_z(x)
   • api_worker_z(x)
   • clock_global
2. Deploy the updated cf-release with the original scale of the above jobs

Impact

• Cloud Foundry components will not attempt to access CCDB until they are able to complete requests.
• Users and clients will get 404 errors instead of 500 errors
• Requests will begin completing successfully when the first instances of the API server come up part way through the deploy
• VMs will need to be created during the deploy which may be slow in some environments

Stop the API jobs

This option prevents access to CCDB by Cloud Foundry components that do not know the updated schema. It also prevents clients from seeing seemingly random errors; they will only see unavailability errors.

Procedure

1. Prior to deploying the new cf-release, perform a bosh stop command on the following jobs.
   • api_z(x)
   • api_worker_z(x)
   • clock_global
2. Deploy the updated cf-release
3. Perform a bosh start command on the jobs that were previously stopped.

Impact

• Cloud Foundry components will not attempt to access CCDB until they are able to complete requests.
• Users and clients will get 404 errors instead of 500 errors
• API requests will not be served until the entire deploy has completed and the API servers have been started. This will create a larger maintenance window than the option to scale down instances.

Failing Deployment Procedure

In the event of a failing deployment related to this migration, operators will have to deploy the previously deployed version of CF Release and restore a CCDB backup. Other components using the database will need to be backed up and restored to roll back to a previously deployed CF release version.

Important Notes

• V3 Service Bindings should be deleted before deploy as the V3 Service Bindings will be deleted without calling out to the broker.
• V2 Apps keep an extra droplet when changing to the most recent droplet. The database record for this droplet will be removed during the migration. We intend to write a background clean up job to remove orphaned blobs from the blobstore in the near future.
• V2 Apps can no longer be moved between spaces.
• V2 Apps can no longer be changed from buildpack to docker and vice versa.
• V2 App usage events recently got "previous" entries for several fields. It is no longer possible to determine previous_package_state so this field has been deprecated.
• V2 Apps running on Diego will now be tagged APP/PROC/WEB/n (where n is the instance number) in application logs sent to syslog drains.
• V2 Apps that are staging during this deploy will likely fail without updating the app. The app should simply be restaged as it will be stuck in the PENDING package state.
• V2 Apps using environment variables beginning with CF_ will be in a bad state after the migration. We plan on releasing a quick follow to fix this.

Job Spec Changes

• pending_packages.expiration_in_seconds has been removed. pending packages will now be expired 5 minutes after the staging timeout.
• pending_packages.frequency_in_seconds has been renamed to pending_droplets.frequency_in_seconds.
• capi.tps.traffic_controller_url now defaults to the internal traffic controller url instead of being dynamically configured to reach the traffic controller through the gorouter. Feel free to remove this from whatever deployment contains TPS (normally Diego deployment). There is an upcoming story to do this automatically for those using the Diego manifest generation scripts.

CAPI Release

• Jobs in CAPI release should use internal URL for traffic controller details

stager

• As a BBS API client, I would like to observe that my LRP instances are not running or my Tasks have failed because there are no cells available matching those placement tags details

Cloud Controller

• CVE-2016-6658: As a CF operator, I expect buildpack urls to be stored and displayed with obfuscated credentials for Audit Events and App Usage Events. details
• CVE-2016-6658: As a CF operator, I expect buildpacks in droplet results to be obfuscated details
• service_plan_guid shouldn't be null in /v2/spaces/:space_guid/service_instances when service plan is private / inactive details
• As a CF operator, I expect to see the CC nginx access and error logs in my syslog data details
• Specifying a buildpack for an app can lead to not being able to GET /v2/apps details
• packages and droplets should be removed from the blobstore when they are expired details
• cc could return a better error when the blobstore is unavailable/misconfigured details
• V3 Alpha
  • V3 links should include protocol and host details
  • As an API user, I would like to list tasks by app with filter for sequence_id details
  • As an app developer, I want tasks for an app to be numbered sequentially details
  • Root of api.some-cloud-foundry.com/v3 should return JSON representation of all resources available in the V3 API. details
  • List tasks for an app optionally shows command and environment_variables details
  • Get a task optionally shows command and environment variables details
  • List tasks never shows command or environment variables details
  • As an app developer, I want tasks for an app to generate their own names if not provided details
  • Root of api.some-cloud-foundry.com should return JSON representation of all APIs available on the Cloud Foundry deployment details

Pull Requests and Issues

• cloudfoundry/cloud_controller_ng#654: handle_local_blobstore.sh cannot create directories details
• cloudfoundry/cloud_controller_ng#659: List all Spaces for the Space Quota Definition has unexplained space_guid parameter details
• cloudfoundry/cloud_controller_ng#663: V3 API cannot delete droplet that has been used for running tasks details
• cloudfoundry/cloud_controller_ng#677: Fix to make validate_add_shared_organization being called details
• cloudfoundry/cloud_controller_ng#679: scripts/which_cf.sh uses CF_RELEASE_DIR environment variable in more places details