Quilt Platform 26.11.0 — Upgrade Guide for Stack Administrators

Quilt Platform 26.11.0

The complete upgrade checklist for stack administrators, from 26.9.x to 26.11.0. Work through Before updating first: several items change what IAM, Lake Formation, and service control policies must allow, and skipping them can fail the update or lock users out of data.

An unmanaged role in the stack's account whose trust policy's principal is the account (arn:aws:iam::<account>:root) rather than the registry's role, even with a condition naming the registry's role, can no longer be assumed after this update, so its users get no credentials: name the stack's RegistryRoleARN output in its trust policy, as Preparing an AWS Role for use with Quilt shows. An unmanaged role in another account registered under one of the account-administration role names that page lists, or with a wildcard or a policy variable in its ARN, is no longer handed out; an edit to a role so named or with such an ARN must give it an allowed role's ARN

Add to the stack any bucket outside it that users open in the catalog, such as one holding package entries, metadata schemas or saved queries, and grant a role any public bucket in the stack its users read without a grant: the catalog no longer reads either, and for an anonymous viewer reads only what every default role is granted. To show a custom s3:// logo on the sign-in page of a stack that requires sign-in, keep it in the service bucket, where the theme editor uploads logos

A bucket's package_handle template that uses anything but username and directory must be rewritten with only those two, or the package dialog pre-fills the username prefix in its place

On the stack-wide package tables, a principal that can set quilt:stack (see Credentials and roles), tag ManagedUserRole or edit its trust policy reads a role's rows, so an account policy must deny iam:CreateRole, iam:TagRole, iam:CreateUser, iam:TagUser and sts:TagSession where ForAnyValue:StringEqualsIgnoreCase matches quilt:stack in aws:TagKeys, and, on each stack's ManagedUserRole, iam:TagRole whatever the key and iam:UpdateAssumeRolePolicy, to every principal but the one CloudFormation deploys each Quilt stack as, its service role where it has one, as the example policy in Reserving the quilt:stack tag key does, and that section also lists the other paths to the key and the policies that close them; no role's trust policy may let SAML or OIDC federation pass quilt:stack as a session tag. Lake Formation's row filters bind Athena, not S3: a principal with S3 read on the service bucket's package-tables/ prefix reads every bucket's rows

A service control policy or other account policy must allow the registry role sts:TagSession with the quilt:role:* and quilt:user keys, or every vend of a managed role's credentials fails. A role assumed from a managed role's vended credentials must now allow sts:TagSession in its trust policy and in every policy that allows sts:AssumeRole on it, ManagedUserRoleExtraPolicies and the managed role's Quilt policies alike, or AssumeRole is refused with "not authorized to perform: sts:TagSession". The stack's tags, which CloudFormation copies onto ManagedUserRole, share the role's 50-tag limit with quilt:stack and the Partner Revenue Measurement tag, so a stack already carrying 49 tags fails this update; a downgrade, or a failed update's rollback, fails managed roles' credential vends for the minutes it takes to replace the registry's tasks

The update that adds the Partner Revenue Measurement tag fails if a stack resource already carries its type's tag limit, counting tags copied from the stack: 50 for most types, the search domain and IAM roles included. It also fails where a tag policy or SCP denies the key aws-apn-id-730278974607

A CloudFormation service role scoped like the installation guide's example needs iam:UpdateAssumeRolePolicy and iam:UntagRole, now in the example, to apply or roll back this release. It also needs lambda:PutFunctionEventInvokeConfig, lambda:GetFunctionEventInvokeConfig, lambda:UpdateFunctionEventInvokeConfig and lambda:DeleteFunctionEventInvokeConfig for every stack's invoke configurations, the stack-wide package tables' among them, and lambda:GetFunctionRecursionConfig, glue:GetDatabase, logs:DescribeIndexPolicies, logs:GetDataProtectionPolicy, lakeformation:DescribeLakeFormationIdentityCenterConfiguration, lakeformation:DescribeResource and lakeformation:ListResources for creating the tables' database, functions and log groups; the example now carries them all. The example role grants every action that adds the Partner Revenue Measurement tag but lacks these actions that remove it, which a failed update's rollback, or a downgrade, uses on resources already tagged: athena:UntagResource, cloudtrail:RemoveTags, ec2:DeleteTags, ecs:UntagResource, elasticloadbalancing:RemoveTags, es:RemoveTags, events:UntagResource, firehose:UntagDeliveryStream, kms:UntagResource, lambda:UntagResource, logs:UntagResource, rds:RemoveTagsFromResource, secretsmanager:UntagResource, servicediscovery:UntagResource, sns:UntagResource, sqs:UntagQueue, ssm:RemoveTagsFromResource, synthetics:UntagResource, wafv2:UntagResource. Without them a rollback can stop in UPDATE_ROLLBACK_FAILED; add them to the role and continue the rollback. CloudFormation may also call ecs:ListTagsForResource, servicediscovery:ListTagsForResource, athena:ListTagsForResource, sns:ListTagsForResource and es:ListTags when it updates these resources' tags, and the example lacks them; grant them before updating

The service bucket reserves qurator-skills/ for the registry, which keeps Qurator skills there: its bucket policy denies object access under that prefix to every other principal, so a backup, replication or bulk copy of the service bucket now stops at it

The registry now calls Lake Formation at every deploy and at every admin's edit of a role's buckets, so Lake Formation must be reachable from the registry's VPC, as Glue, Athena, IAM and STS are, and allowed by the account's service control policies. Where it is not, admins' edits of policies and managed roles fail, and no user can read the stack-wide package tables until the registry's migration, run at each release's update, first reaches Lake Formation; after that, the registry's failure to call it takes no reads from users, but no role reads the rows of a bucket first added to the stack while that failure lasts, until the migration next reaches it

Updating to this release creates the stack-wide package tables, and the registry's migration task fills them from every registered bucket without holding up the update; the next release's update completes a fill that stopped short. On an account that enforces Lake Formation the tables need EnableLakeFormationGrants set to Enabled, and a managed role already reading more than 100 buckets reads nothing of them until its buckets are split across roles

Updating a stack to this release, or adding a bucket to a stack that runs it, fills the stack-wide package tables without the revisions and tags of packages whose name begins with an underscore in either part, such as _team/data or team/_data, though with their manifests and entries, and a later deploy does not add them; the tables record a later revision of such a package, with the latest tag it moves. The documentation's "Querying packages with Iceberg tables" page says which of them the per-bucket tables hold

The stack-wide package tables' two functions reserve 3 concurrent executions between them, so an update to this release fails on an account that cannot reserve them above Lambda's unreserved-concurrency minimum. The functions that forward S3's native EventBridge events and package-revision events, also new on every stack, reserve 10 and 5 more, 18 in all

Updating to this release creates a Glue database named quilt__ and the stack's name in lower case, every character outside a-z and 0-9 replaced by _. The update fails at its creation and rolls back where a database of that name exists, as it does for the second of two stacks in one account and region whose names differ only in case

Updating to this release deletes the stack's Voila resources, among them its EC2 instance, ECS service and load balancer rule. A stack that passes VoilaVersion or VoilaAMI explicitly, in a CLI parameter list or in Terraform, must stop passing them in the same update that moves it to this release: CloudFormation rejects an update that passes a parameter the template no longer has

The per-bucket Iceberg tables are deprecated: they keep working in this release, and a later release removes them in favor of the stack-wide package tables, which carry the bucket as their registry column

A rollback of the update to this release, a downgrade past it or the stack's deletion removes the stack-wide package tables but leaves their package-tables/ prefix registered with Lake Formation and its files in the service bucket. The documentation's "Querying packages with Iceberg tables" page gives the steps that clear them, before a downgrade or deletion and before an update that failed is retried

The upgrade signs every user out and revokes Connect access tokens at once, but leaves API keys and passwords as they are, and AWS credentials issued before it stay valid for up to an hour: from an hour after the upgrade, review API keys with quilt3.admin.api_keys, users and admins, and the SSO configuration's mappings, and remove any you do not recognize, since a key or an account made with a stolen session outlives the sign-out; have users who may have been exposed change their passwords

Admins are advised to press "Revoke all API keys" in Admin → Settings → Security once after upgrading to this release, after telling key owners to expect to issue a new one

Users' session tokens the S3 proxy already logged stay valid until their sessions end, and the log streams the S3 proxy wrote before the upgrade still hold them: delete those, prefixed s3-proxy/ in the log group named after the stack

Task-definition revisions from before this release keep the AdminPassword value they were given, readable to any principal allowed ecs:DescribeTaskDefinition: CloudFormation deregisters a superseded revision but does not delete it. If the admin still signs in with that password, reset it from the catalog's sign-in page, since changing AdminPassword does nothing to an existing admin; until then, deleting the inactive revisions (aws ecs delete-task-definitions) stops it being read there. On a stack with sign-in through an SSO provider, those revisions hold an empty AdminPassword, so there is nothing to clean

Task-definition revisions from before this release keep the database password and the SSO client secrets in plain text, and DescribeTaskDefinition still returns them: each stack update leaves the previous revision INACTIVE instead of deleting it. Rotate the database password (the one in DBUrl, or DBPassword) and the SSO client secrets; an admin who cannot rotate yet can delete the INACTIVE revisions (aws ecs delete-task-definitions), which removes readable copies of a still-valid value but undoes no read already made

On a stack deployed with Terraform from quiltdata/iac, Terraform generates the database password and builds DBUrl from it, so once the stack runs this release, rotate the password with terraform apply -replace='<address>', where <address> is the stack's own line of terraform state list | grep module.db.module.db.random_password, and redeploy the services as below right after it: their new database connections fail until then, so plan a maintenance window. A reset in the RDS console, which Terraform does not detect, or a stack update that fails during the apply leaves DBUrl and the stack's secret on the old password, so the services' new connections fail until the same -replace and redeploy run

A new password for the same database needs no migrations, but a database that needs them, such as an empty one, one restored from a snapshot taken before this release, or the new one a DBUser change creates on a stack with its own database, needs the migration task run once before the services are redeployed. Run it with aws ecs run-task --cluster <stack> --task-definition <stack>-registry-migration --launch-type FARGATE --network-configuration <config>, where <config> is the networkConfiguration that aws ecs describe-services --cluster <stack> --services <stack>-bulk-scanner returns

Once a stack runs this release, an update that changes only DBUrl, DBPassword or an SSO client secret changes that value's Secrets Manager secret but no task definition, so it neither restarts running tasks nor reruns the database migrations: redeploy <stack>-registry and, for the database, <stack>-bulk-scanner and, where Qurator is enabled, <stack>-batch-summarizer, with aws ecs update-service --cluster <stack> --service <service> --force-new-deployment. A service whose own task definition changes in the same update, as these do in an upgrade that ships a new registry image, needs no extra step

For the Help panel's chat with Quilt support and sales, the catalog loads HubSpot's script from js.hs-scripts.com on every page. Some customer variants are exempt, and an admin can turn the chat widget off from Admin → Settings

Changing pkg_prefix does not move existing entry packages; each entry's package is created under the new name the next time it is packaged. If the bucket's Quilt workflow sets a handle_pattern, it must accept the new namespaces

For the full picture of what 26.11.0 ships, see the release email, or write support@quilt.bio for the complete notes. Upgrade mechanics live in the upgrade documentation.