💥 ALERT!!
This repo will soon be relocating to GoogleCloudPlatform as we better organize these code samples! Stay tuned as more info is coming soon.

Modernizing Google Cloud serverless compute applications

To the latest Cloud services and serverless platforms

This is the corresponding repository to the Serverless Migration Station video series whose goal is to help users on a Google Cloud serverless compute platform modernize to newer Cloud products or other serverless compute platforms. Each modernization migration aims to feature a video, codelab (self-paced, hands-on tutorial), and code samples. The content initially focuses on App Engine and Google's earliest Cloud users. Read more about the codelabs in this announcement as well as this one introducing the video series.

Google App Engine (Standard) has undergone significant changes between the legacy and next generation platforms. To address this, we've created a set of codelabs (free, online, self-paced, hands-on tutorials) and corresponding videos (when available) to show developers how to perform individual migrations they can apply to modernize their apps for the latest runtimes, with this repo managing the samples from those codelabs. Codelab content typically falls into one of these three topics:

Migrate from a legacy App Engine service to a Cloud equivalent product
Migrate to a newer Cloud product, service, or API client library
Migrate to another or newer Google Cloud serverless compute platform

Each codelab begins with a "START" code base then walks developers through that migration step, resulting in a "FINISH" repo. If you make any mistakes along the way, you can always go back to START or compare your code with what's in the FINISH folder to see the differences. We also want to help you port to the Python 3 runtime, so some codelabs contain a bonus section for that purpose.

NOTE: These migrations are typically aimed at Python 2 users

Python 3.x App Engine users: You're already on the next-gen platform, so only look for non-legacy service migrations

Python 2.5 App Engine users: to revive apps from the original 2.5 runtime, deprecated in 2013 and shutdown in 2017, you must migrate from db to ndb and get those apps running on Python 2.7 before attempting these migrations.

Prerequisites

A Google account (Google Workspace/G Suite accounts may require administrator approval)
A Google Cloud (GCP) project with an active billing account
Familiarity with operating system terminal/shell commands
Familiarity with developing & deploying Python 2 apps to App Engine
General skills in Python 2 and 3

Cost

App Engine, Cloud Functions, and Cloud Run are not free services. While you may not have needed to enable billing in App Engine's early days, all applications now require an active billing account backed by a financial instrument (usually a credit card). Don't worry, App Engine (and other GCP products) still have an "Always Free" tier and as long as you stay within those limits, you won't incur any charges. Also check the App Engine pricing and quotas pages for more information.

Furthermore, deploying to GCP serverless platforms incur minor build and storage costs. Cloud Build has its own free quota as does Cloud Storage. For greater transparency, Cloud Build builds your application image which is than sent to the Cloud Container Registry, or Artifact Registry, its successor; storage of that image uses up some of that (Cloud Storage) quota as does network egress when transferring that image to the service you're deploying to. However you may live in region that does not have such a free tier, so be aware of your storage usage to minimize potential costs. (You may look at what storage you're using and how much, including deleting build artifacts via your Cloud Storage browser.)

Why

In App Engine's early days, users wanted Google to make the platform more flexible for developers and make their apps more portable. As a result, the team made significant changes to its 2nd-generation service which launched in 2018. As a result, there are no longer any built-in services, allowing users to select from standalone GCP products or best-of-breed 3rd-party tools used by the broader community. Summary:

Legacy platform: Python 2 only, legacy built-in services
Next generation: Python 3 only, use standalone services, flexible platform, some* built-in services available

^* see Legacy services section below

While the 2nd-gen platform is more flexible, users of the legacy platform have two challenges:

Migrate to unbundled/standalone services
Porting to a modern language release

Neither upgrade may be particularly straightforward and can only be done serially. On top of this, direct replacements are not available for all formerly built-in services; alternatives come in 3 flavors:

Direct replacement: Legacy services which matured into their own Cloud products (e.g., App Engine Datastore is now Cloud Datastore)
Partial replacement: Some aspects of legacy services (e.g., Cloud Tasks supports App Engine push tasks; for pull tasks, Cloud Pub/Sub is recommended; use of Cloud MemoryStore with REDIS as an alternative for Memcache)
No replacement: No direct replacement available, so third-party or other tools recommended (e.g., Search, Images, Users, Email)

These are the challenges developers are facing, so the purpose of this content is to reduce the friction in this process and make things more prescriptive. Review the runtimes chart to see the legacy services and current migration recommendation. The migration guide overview has more information.

NOTE: App Engine (Flexible) is a next-gen service but is not within the scope of these tutorials. Curious developers can compare App Engine Standard vs. Flexible to find out more. Also, many of the Flexible use cases can now be handled by Cloud Run.

Progression (what order to do things)

"START" and "FINISH" repo folders

All codelabs begin with code in a START repo folder and end with code in a FINISH folder, implementing a single migration. Upon completion, users should confirm their code (for the most part) matches what's in the FINISH folder. The baseline migration sample app (Module 0; link below) is a barebones Python 2.7 App Engine app that uses the webapp2 web framework plus the ndb Datastore library.

With Module 0 as the STARTing point, the Module 1 codelab migrates from the webapp2 web framework to Flask, FINISHing at code matching the Module 1 repo.
Next, STARTing with the Module 1 application code (yours or ours), Module 2 migrates from ndb to Cloud NDB, ending with code matching the (Module 2) FINISH repo folder. There's also has a bonus migration to Python 3, resulting in another FINISH repo folder, this one deployed on the next-generation platform.
Your Python 2 apps may be using other built-in services like Task Queues or Memcache, so additional migration modules follow, some more optional than others, and not all are available yet (keep checking back here for updates).

The order of migrations

Beyond Module 2, with some exceptions, there is no specific order of what migrations modules to tackle next. It depends on your needs (and your applications'). However, there are related migrations where one or more modules must be completed beforehand. This table attempts to put an order on module subsets.

Topic	Module ordering	Description
Baseline	0 ⇒ 1	Not a migration but a description of the baseline application (review this material before doing any migrations)
Web framework	1 ⇒ everything else	Current App Engine runtimes do not come with a web framework, so this must be the first migration performed. All migrations below can be performed after this one.
Datastore	2 [⇒ 3 [⇒ 6]]	Moving off App Engine `ndb` makes your apps more portable, so the Module 2 Cloud NDB migration is recommended. Module 3: Migrating to Cloud Datastore (Firestore in Datastore mode) is optional and only recommended if you have other code using Cloud Datastore. Module 6: Migrating to Cloud Firestore (Native mode) is generally not recommended unless you must have the Firebase features it has, and those features will eventually be integrated into Cloud Datastore.
(Push) Task Queues	[7 ⇒] 8 [⇒ 9]	Moving off App Engine `taskqueue` makes your apps more portable, so the Module 8 Cloud Tasks migration is recommended for those using push tasks. Those unfamiliar with push tasks should do Module 7 first to add push tasks to the sample app. Module 9: Migrating to Cloud Datastore (Firestore in Datastore mode), Cloud Tasks (v2), and Python 3 is optional and only recommended if you have other code using Cloud Datastore and considering upgrading to Python 3.
Memcache	[12 ⇒] 13	Moving off App Engine `memcache` makes your apps more portable, so the Module 13 Cloud Memorystore (for Redis) migration is recommended for those using `memcache`. Those unfamiliar with `memcache` should do Module 12 first to add its usage to the sample app.
Cloud Functions	11	Cloud Functions does not support Python 2, so after the Module 1 migration, you need to upgrade your app to Python 3 before attempting this migration, recommended if you have a very small App Engine app, or it has only one function/feature.
Cloud Run	4 or 5	Module 4 covers migrating to Cloud Run with Docker. Those unfamiliar with containers or do not wish to create/maintain a `Dockerfile` should do Module 5. Those doing Module 4 will get additional information about Cloud Run in Module 5 not covered in Module 4.

Migration modules

The table below summarizes migration module resources currently available along with a more detailed table of contents below. Be sure to check back for updates as more resources are planned.

Summary table

Module	Topic	Video	Codelab	START here	FINISH here
0	Baseline app	link	N/A (no tutorial; just review the code)	N/A	Module 0 code (2.x)
1	Migrate to Flask	link	link	Module 0 code (2.x)	Module 1 code (2.x) & code (3.x)
2	Migrate to Cloud NDB	link	link	Module 1 code (2.x)	Module 2 code (2.x) & code (3.x)
3	Migrate to Cloud Datastore	link	link	Module 2 code (2.x) & code (3.x)	Module 3 code (2.x) & code (3.x)
4	Migrate to Cloud Run with Docker	link	link	Module 2 code (2.x) & Module 3 code (3.x)	Module 4 code (2.x) & code (3.x)
5	Migrate to Cloud Run with Buildpacks	link	link	Module 2 code (3.x)	Module 5 code (3.x)
6	Migrate to Cloud Firestore	N/A	N/A	Module 3 code (3.x)	no work required; Datastore upgrade automatic
7	Add App Engine `taskqueue` push tasks	link	link	Module 1 code (2.x)	Module 7 code (2.x) & code (3.x)
8	Migrate to Cloud Tasks	link	link	Module 7 code (2.x)	Module 8 code (2.x)
9	Migrate to Python 3, Cloud Datastore & Cloud Tasks v2	TBD	TBD	Module 8 code (2.x)	Module 9 code
10	Migrate Datastore/Firestore data to another project	TBD	N/A	N/A	TBD
11	Migrate to Cloud Functions	TBD	link	Module 2 code (3.x)	Module 11 code (3.x)
12	Add App Engine `memcache`	TBD	link	Module 1 code (2.x)	Module 12 code (2.x) & code (3.x)
13	Migrate to Cloud Memorystore	TBD	TBD	Module 12 code (2.x) & code (3.x)	Module 13 code (2.x) & code (3.x)

If there is a logical codelab to do immediately after completing one, they will be designated as NEXT. Other recommended codelabs will be listed as RECOMMENDED, and the more optional ones will be labeled as OTHERS (and usually in some kind of priority order).

Module 1 codelab: Migrate from webapp2 to Flask
- Required migration (can also pick your own framework)
  - webapp2 does not do routing thus unsupported by App Engine (even though a 3.x port exists)
- Python 2 only
  - START: Module 0 code - Baseline (2.x)
  - FINISH: Module 1 code - Framework (2.x)
- NEXT:
  - Module 2 - migrate to Cloud NDB
Module 2 codelab: Migrate from App Engine ndb to Cloud NDB
- Required migration
  - Migration to Cloud NDB which is supported by Python 3 and the next-gen platform.
- Python 2
  - START: Module 1 code - Framework (2.x)
  - FINISH: Module 2 code - Cloud NDB (2.x)
- Codelab bonus port to Python 3.x
  - FINISH: Module 2 code - Cloud NDB (3.x)
- RECOMMENDED:
  - Module 7 - add App Engine (push) tasks
- OTHERS (somewhat priority order):
  - Module 11 - migrate to Cloud Functions
  - Module 5 - migrate to Cloud Run container with Cloud Buildpacks
  - Module 4 - migrate to Cloud Run container with Docker
  - Module 3 - migrate to Cloud Datastore
Module 7 codelab: Add App Engine (push) Task Queues to App Engine ndb Flask app
- Not a migration: add GAE Task Queues to prepare for migration to Cloud Tasks
- Python 2
  - START: Module 1 code - Framework (2.x)
  - FINISH: Module 7 code - GAE Task Queues (2.x)
- NEXT: Module 8 - migrate App Engine push tasks to Cloud Tasks
Module 8 codelab: Migrate from App Engine (push) Task Queues to Cloud Tasks v1
- Required migration
  - Migration to Cloud Tasks which is supported by Python 3 and the next-gen platform.
  - Note this is only push tasks... pull tasks will be handled in a different codelab.
- Python 2
  - START: Module 7 code - GAE Task Queues (2.x)
  - FINISH: Module 8 code - Cloud Tasks (2.x)
- NEXT: Module 9 - migrate to Python 3 and Cloud Datastore
Module 9 codelab (TBD): Migrate a Python 2 Cloud NDB & Cloud Tasks (v1) app to a Python 3 Cloud Datastore & Cloud Tasks (v2) app
- Optional migrations
  - Migrating to Python 3 is not required but recommended as Python 2 has been sunset
  - Migrating to Cloud Datastore is optional as Cloud NDB works on 3.x
- Python 2
  - START: Module 8 code - Cloud Tasks (2.x)
- Python 3
  - FINISH: TBD
- RECOMMENDED:
  - Module 11 - migrate to Cloud Functions
  - Module 5 - migrate to Cloud Run container with Cloud Buildpacks
  - Module 4 - migrate to Cloud Run container with Docker
Module 4 codelab: Migrate from App Engine to Cloud Run with Docker
- Optional migration
  - "Containerize" your app (migrate your app to a container) with Docker
- Python 2
  - START: Module 2 code - Cloud NDB (2.x)
  - FINISH: Module 4 code - Cloud Run - Docker 3.x (2.x)
- Python 3
  - START: Module 3 code - Cloud Datastore (3.x)
  - FINISH: Module 4 code - Cloud Run - Docker (3.x)
- RECOMMENDED:
  - Module 5 - migrate to Cloud Run container with Cloud Buildpacks
- OTHER OPTIONS (in somewhat priority order):
  - Module 7 - add App Engine (push) tasks
  - Module 11 - migrate to Cloud Functions
Module 5 codelab: Migrate from App Engine to Cloud Run with Cloud Buildpacks
- Optional migration
  - "Containerize" your app (migrate your app to a container) with...
  - Cloud Buildpacks which lets you containerize your app without Dockerfiles
- Python 3 only
  - START: Module 2 code - Cloud NDB (3.x)
  - FINISH: Module 5 code - Cloud Run - Buildpacks 3.x (3.x)
- RECOMMENDED:
  - Module 4 - migrate to Cloud Run container with Docker
- OTHER OPTIONS (in somewhat priority order):
  - Module 7 - add App Engine (push) tasks
  - Module 11 - migrate to Cloud Functions
Module 3 codelab: Migrate from Cloud NDB to Cloud Datastore
- Optional migration
  - Recommended only if using Cloud Datastore elsewhere (GAE or non-App Engine) apps
  - Helps w/code consistency & reusability, reduces maintenance costs
- Python 2
  - START: Module 2 code - Cloud NDB (2.x)
  - FINISH: Module 3 code - Cloud Datastore (2.x)
- Python 3
  - START: Module 2 code - Cloud NDB (3.x)
  - FINISH: Module 3 code - Cloud Datastore (3.x)
- RECOMMENDED:
  - Module 7 - add App Engine (push) tasks
- OTHER OPTIONS (in somewhat priority order):
  - Module 11 - migrate to Cloud Functions
  - Module 5 - migrate to Cloud Run container with Cloud Buildpacks
  - Module 4 - migrate to Cloud Run container with Docker
Module 11 codelab: Migrate from App Engine to Cloud Functions
- Optional migration
  - Recommende for small apps or for breaking up large apps into multiple microservices
- Python 3 only
  - START: Module 2 code - Cloud NDB (3.x)
  - FINISH: Module 11 code - Cloud Functions (3.x)
- RECOMMENDED:
  - Module 7 - add App Engine (push) tasks
- OTHER OPTIONS (in somewhat priority order):
  - Module 5 - migrate to Cloud Run container with Cloud Buildpacks
  - Module 4 - migrate to Cloud Run container with Docker
  - Module 3 - migrate to Cloud Datastore

Considerations for mobile developers

If your original app users does not have a user interface, i.e., mobile backends, etc., but still uses webapp2 for routing, some migration must still be completed. Your options:

Migrate to Flask (or another) web framework but keep app on App Engine
Use Cloud Endpoints or Cloud API Gateway for your mobile endpoints
Break-up your monolithic app to "microservices" and migrate your app to either:
- Google Cloud Functions
- Firebase mobile & web app platform (and Cloud Functions for Firebase [customized for Firebase])

Canonical code samples

This repo, along with corresponding codelabs & videos are complementary to the official docs & code samples.
- The official Python 2 to 3 migration documentation
- Canonical migration code samples repo
  - Example: GAE ndb to Cloud NDB (similar to Module 2)
  - Example: GAE taskqueue to Cloud Tasks (similar to Module 8)

Accessing legacy services in second generation

Many legacy App Engine first generation platform (Python 2, Java 8, PHP 5, and Go 1.11 & older) services are available (as of Sep 2021 for second generation runtimes (Python 3, Java 11, PHP 7, and Go 1.12 & newer) in a public preview. There are no videos or codelabs yet, however the Module 1 Flask migration using App Engine ndb Python 2 sample is available in Python 3 if you have access. Similarly, Python 3 editions are also available for Modules 7 and 12 which add usage of App Engine taskqueue and memcache, respectively.

References

App Engine Migration
- Migrate from Python 2 to 3
- Migrate from App Engine ndb to Cloud NDB (Module 2)
- Migrate from App Engine taskqueue to Cloud Tasks (Modules 7-9)
- Migrate from App Engine db to ndb ("Module -1"; only for reviving "dead" Python 2.5 apps for 2.7)
- Community contributed migration samples
Python App Engine
Google Cloud Platform (GCP)

coreonyx / migrate-python2-appengine