arrow_back

Online Data Migration to Cloud Spanner using Striim

Teilnehmen Anmelden
Test and share your knowledge with our community!
done
Get access to over 700 hands-on labs, skill badges, and courses

Online Data Migration to Cloud Spanner using Striim

Lab 1 Stunde universal_currency_alt 5 Guthabenpunkte show_chart Mittelstufe
Test and share your knowledge with our community!
done
Get access to over 700 hands-on labs, skill badges, and courses

This lab was developed with our partner, Striim. Your personal information may be shared with Striim, the lab sponsor, if you have opted-in to receive product updates, announcements, and offers in your Account Profile.

GSP703

Google Cloud self-paced labs logo

Overview

In this lab you will learn how to migrate a Cloud SQL for MySQL database to Cloud Spanner using Google Cloud's Data Migration Partner, Striim.

Striim is a comprehensive streaming ETL platform that enables online data migrations and continuous streaming replication from on-premises and cloud data sources to Google Cloud data services through a graphical drag and drop interface. The following figure shows Striim's high level architecture.

Striim platform architecture diagram

Note: This lab will focus on the implementation of a continuous migration from MySQL to Cloud Spanner, and is not an explanation of database migration or database replication, or why you might want to migrate your underlying database.

A diagram of the data flow from Cloud SQL for MySQL to Cloud Spanner via Striim.

The figure shows the various services that you will use and create as well as the data flow from Cloud SQL for MySQL to Cloud Spanner via Striim that you will implement.

What you will learn

In this hands-on-lab you learn how to:

  • Access and use an existing Cloud SQL for MySQL database.
  • Access and use an existing Cloud Spanner database.
  • Deploy Striim through the Google Cloud Marketplace and set it up.
  • Use Striim to read from a source Cloud SQL for MySQL database.
  • Use Striim to continuously replicate from Cloud SQL for MySQL to Cloud Spanner.

Setup and requirements

Before you click the Start Lab button

Read these instructions. Labs are timed and you cannot pause them. The timer, which starts when you click Start Lab, shows how long Google Cloud resources will be made available to you.

This hands-on lab lets you do the lab activities yourself in a real cloud environment, not in a simulation or demo environment. It does so by giving you new, temporary credentials that you use to sign in and access Google Cloud for the duration of the lab.

To complete this lab, you need:

  • Access to a standard internet browser (Chrome browser recommended).
Note: Use an Incognito or private browser window to run this lab. This prevents any conflicts between your personal account and the Student account, which may cause extra charges incurred to your personal account.
  • Time to complete the lab---remember, once you start, you cannot pause a lab.
Note: If you already have your own personal Google Cloud account or project, do not use it for this lab to avoid extra charges to your account.

How to start your lab and sign in to the Google Cloud console

  1. Click the Start Lab button. If you need to pay for the lab, a pop-up opens for you to select your payment method. On the left is the Lab Details panel with the following:

    • The Open Google Cloud console button
    • Time remaining
    • The temporary credentials that you must use for this lab
    • Other information, if needed, to step through this lab
  2. Click Open Google Cloud console (or right-click and select Open Link in Incognito Window if you are running the Chrome browser).

    The lab spins up resources, and then opens another tab that shows the Sign in page.

    Tip: Arrange the tabs in separate windows, side-by-side.

    Note: If you see the Choose an account dialog, click Use Another Account.
  3. If necessary, copy the Username below and paste it into the Sign in dialog.

    {{{user_0.username | "Username"}}}

    You can also find the Username in the Lab Details panel.

  4. Click Next.

  5. Copy the Password below and paste it into the Welcome dialog.

    {{{user_0.password | "Password"}}}

    You can also find the Password in the Lab Details panel.

  6. Click Next.

    Important: You must use the credentials the lab provides you. Do not use your Google Cloud account credentials. Note: Using your own Google Cloud account for this lab may incur extra charges.
  7. Click through the subsequent pages:

    • Accept the terms and conditions.
    • Do not add recovery options or two-factor authentication (because this is a temporary account).
    • Do not sign up for free trials.

After a few moments, the Google Cloud console opens in this tab.

Note: To view a menu with a list of Google Cloud products and services, click the Navigation menu at the top-left. Navigation menu icon

Activate Cloud Shell

Cloud Shell is a virtual machine that is loaded with development tools. It offers a persistent 5GB home directory and runs on the Google Cloud. Cloud Shell provides command-line access to your Google Cloud resources.

  1. Click Activate Cloud Shell Activate Cloud Shell icon at the top of the Google Cloud console.

When you are connected, you are already authenticated, and the project is set to your Project_ID, . The output contains a line that declares the Project_ID for this session:

Your Cloud Platform project in this session is set to {{{project_0.project_id | "PROJECT_ID"}}}

gcloud is the command-line tool for Google Cloud. It comes pre-installed on Cloud Shell and supports tab-completion.

  1. (Optional) You can list the active account name with this command:
gcloud auth list
  1. Click Authorize.

Output:

ACTIVE: * ACCOUNT: {{{user_0.username | "ACCOUNT"}}} To set the active account, run: $ gcloud config set account `ACCOUNT`
  1. (Optional) You can list the project ID with this command:
gcloud config list project

Output:

[core] project = {{{project_0.project_id | "PROJECT_ID"}}} Note: For full documentation of gcloud, in Google Cloud, refer to the gcloud CLI overview guide.

Before you begin

In order to work on the database migration some initial configuration is required:

  1. In Cloud Shell, set the default compute zone you wish to deploy your database and compute resources in. For more information, please see Geography and Regions. For the purposes of this lab you will use .
gcloud config set compute/zone "{{{project_0.default_zone | Zone}}}" export COMPUTE_ZONE="{{{project_0.default_zone | Zone}}}"
  1. Verify that MySQL has been setup. Execute the following commands:
CSQL_NAME=striim-sql-src CSQL_USER_NAME=striim-user CSQL_USER_PWD=striim-usr-password CSQL_ROOT_PWD=striim-root-password
  1. Next, with the local variables you just made, connect via executing the following command. It will prompt you for a password, which is the CSQL_USER_PWD you just created, striim-usr-password.
gcloud sql connect $CSQL_NAME --user=$CSQL_USER_NAME
  1. Once you're in, start with these commands:
USE striimdemo; SELECT 1 FROM DUAL;
  1. Then, load some transactions into it at the mysql> prompt:
USE striimdemo; CREATE TABLE ORDERS (ORDER_ID Integer, ORDER_DATE VARCHAR(50), ORDER_MODE VARCHAR(8), CUSTOMER_ID Integer, ORDER_STATUS Integer, ORDER_TOTAL Float, SALES_REP_ID Integer, PROMOTION_ID Integer, PRIMARY KEY (ORDER_ID)); INSERT INTO ORDERS (ORDER_ID, ORDER_DATE, ORDER_MODE, CUSTOMER_ID, ORDER_STATUS, ORDER_TOTAL, SALES_REP_ID, PROMOTION_ID) VALUES (1001, 1568927976017, 'In-Store', 1001, 9, 34672.59, 331, 9404); INSERT INTO ORDERS (ORDER_ID, ORDER_DATE, ORDER_MODE, CUSTOMER_ID, ORDER_STATUS, ORDER_TOTAL, SALES_REP_ID, PROMOTION_ID) VALUES (1002, 1568928036017, 'In-Store', 1002, 1, 28133.14, 619, 2689); INSERT INTO ORDERS (ORDER_ID, ORDER_DATE, ORDER_MODE, CUSTOMER_ID, ORDER_STATUS, ORDER_TOTAL, SALES_REP_ID, PROMOTION_ID) VALUES (1003, 1568928096017, 'CompanyB', 1003, 1, 37367.95, 160, 30888); INSERT INTO ORDERS (ORDER_ID, ORDER_DATE, ORDER_MODE, CUSTOMER_ID, ORDER_STATUS, ORDER_TOTAL, SALES_REP_ID, PROMOTION_ID) VALUES (1004, 1568928156017, 'CompanyA', 1004, 1, 7737.02, 362, 89488); INSERT INTO ORDERS (ORDER_ID, ORDER_DATE, ORDER_MODE, CUSTOMER_ID, ORDER_STATUS, ORDER_TOTAL, SALES_REP_ID, PROMOTION_ID) VALUES (1005, 1568928216017, 'CompanyA', 1005, 9, 15959.91, 497, 78454); INSERT INTO ORDERS (ORDER_ID, ORDER_DATE, ORDER_MODE, CUSTOMER_ID, ORDER_STATUS, ORDER_TOTAL, SALES_REP_ID, PROMOTION_ID) VALUES (1006, 1568928276017, 'In-Store', 1006, 1, 82531.55, 399, 22488); INSERT INTO ORDERS (ORDER_ID, ORDER_DATE, ORDER_MODE, CUSTOMER_ID, ORDER_STATUS, ORDER_TOTAL, SALES_REP_ID, PROMOTION_ID) VALUES (1007, 1568928336017, 'CompanyA', 1007, 7, 52929.61, 420, 66256); INSERT INTO ORDERS (ORDER_ID, ORDER_DATE, ORDER_MODE, CUSTOMER_ID, ORDER_STATUS, ORDER_TOTAL, SALES_REP_ID, PROMOTION_ID) VALUES (1008, 1568928396017, 'Online', 1008, 1, 26912.56, 832, 7262); INSERT INTO ORDERS (ORDER_ID, ORDER_DATE, ORDER_MODE, CUSTOMER_ID, ORDER_STATUS, ORDER_TOTAL, SALES_REP_ID, PROMOTION_ID) VALUES (1009, 1568928456017, 'CompanyA', 1009, 1, 97706.08, 124, 12185); INSERT INTO ORDERS (ORDER_ID, ORDER_DATE, ORDER_MODE, CUSTOMER_ID, ORDER_STATUS, ORDER_TOTAL, SALES_REP_ID, PROMOTION_ID) VALUES (1010, 1568928516017, 'CompanyB', 1010, 1, 47539.16, 105, 17868);
  1. Verify that Cloud Spanner instance has been setup with a database and table defined. In the Google Cloud console, select Navigation menu > Spanner from the left pane.

  2. Click on the instance striim-spanner-demo, and then click on the database striimdemo. Check that the table orders is present.

  3. Click on the table orders, and then the tab DATA. The table should be empty.

The Data tabbed page, which desplays an empty table.

  1. Click on the SCHEMA and you should see that the schema has been defined.

The Schema tabbed page, which includes information such as the order ID, status, and promotion ID.

Task 1. Set up the Striim software

Next, set up an instance of the Striim server through the Google Cloud Marketplace.

  1. Navigate to the Striim offering on the Google Cloud Marketplace. If a list of Google Cloud projects is displayed then select the Project ID of this lab.

Open Striim in the marketplace

Alternatively you can search in the Marketplace using the direct link.

  1. Click the button marked Get Started.

  2. Check the box to agree to the Terms and agreements and click Agree.

  3. Click Deploy. You will be presented with configuration options for the Striim server.

  4. Change the zone to .

  5. Choose whether to authorize information sharing with Striim.

  6. Leave all other settings as is, and click the Deploy button.

Note: Marketplace solutions typically come with various resources that launch together in order to support the software. Be sure to review the monthly billing estimate before launching Striim in your production account.

You will be redirected to Deployment Manager page, once the solution is provisioned, the deployment details will show.

  1. Take note of the name of the VM ("Instance" in the example screenshot below) and the zone it is deployed in ("Instance zone").

The Striim page, which includes the highlighted instance URL and instance zone.

  1. If you didn't already exit from the mysql> prompt, type exit to go back to Cloud Shell. Once you're back in, execute this command replacing the VM name [STRIIM VM NAME] and Zone [STRIIM VM ZONE] from the deployment details pane:
STRIIMVM_NAME=[STRIIM VM NAME] STRIIMVM_ZONE=[STRIIM VM ZONE]
  1. Now in order to allow Striim to communicate with Cloud SQL, you need to add the Striim server's IP address to allow it to connect. Execute the following command and acknowledge that you want to continue with Y when prompted:
gcloud sql instances patch $CSQL_NAME --authorized-networks=$(gcloud compute instances describe $STRIIMVM_NAME --format='get(networkInterfaces[0].accessConfigs[0].natIP)' --zone=$STRIIMVM_ZONE)
  1. In the Deployment Manager page, click Visit the site. This will open up the Striim configuration wizard in a new browser tab. If there is a redirect notice, you can safely ignore it and click through the link provided.

  2. Accept the EULA by clicking Accept Striim EULA and Continue.

  3. Configure your Striim installation:

Property Value (type value or select option as specified)
First name Example
Last name User
Email user@example.com
Company Name Example
Cluster name my_cluster
Admin password my_admin_password
Sys user password my_cluster_password
Striim Key Store password my_cluster_password
Note: Do not use spaces or other punctuation in the form.

Be sure to note the cluster name and the cluster and administrator passwords, as you will be using them later in this lab.

  1. Click Save and Continue.

  2. In the next screen will ask for license details. Leave the field empty and click Save and Continue.

  3. Click Launch and then click the green Log In button.

  4. Log in using the admin user and the administrator password my_admin_password. This will take you to the Striim administrator console (screen shown below), and you are now ready to connect to MySQL.

Note: The "admin" user is a built-in user that must be used in order for login to succeed.

The Striim administrator console, which includes the Dashboard, Apps, and Source Preview tiles.

Click Check my progress to verify the objective. Set up the Striim software

Task 2. Set up Connector/J

You will now follow the Striim online documentation and use the MySQL Connector/J to connect Striim to your Cloud SQL instance.

  1. In the Deployment Manager page for your Striim instance, click the SSH button located next to the blue Visit the site button. This will open up a new window and will automatically SSH into the virtual machine.

  2. Download Connector/J to the VM and unpack it:

sudo curl -LO https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-8.0.29.tar.gz sudo tar -xvzf mysql-connector-java-8.0.29.tar.gz
  1. Copy the file to the Striim library path, allow it to be executable, and change ownership of it:
sudo cp mysql-connector-java-8.0.29/mysql-connector-java-8.0.29.jar /opt/striim/lib sudo chmod +x /opt/striim/lib/mysql-connector-java-8.0.29.jar sudo chown striim /opt/striim/lib/mysql-connector-java-8.0.29.jar

Click Check my progress to verify the objective. Setting up Connector/J

  1. Run the following commands to create the user directory and give the user permissions to access it:
export user_id=`whoami` sudo mkdir /home/$user_id sudo chown -R $user_id /home/$user_id sudo chmod -R u+rX /home/$user_id
  1. You now need to restart the Striim server to recognize the new library. Do this in the same SSH session by running the following:
sudo systemctl stop striim-node sudo systemctl stop striim-dbms sudo systemctl start striim-dbms sudo systemctl start striim-node
  1. Go back to the browser window for the Striim administration console, refresh the window, and log in using the admin user and password. The administration console will take a few minutes to refresh after the restart. If the administration console does not reload on the first try, keep refreshing until the administration console comes up.

If the administration console does not come up, use the site address, but with port 9080 instead of 9070, i.e. http://[IP_ADDRESS]:9080/

Note: The admin user is a built-in user that must be used in order for login to succeed at the Striim administration console.

Task 3. Create a Cloud Spanner access key

Striim requires an access key so that it can connect to Cloud Spanner. Here are the required steps you need to do:

  1. Create a service account for Striim to use to connect to Cloud Spanner. Run the below command in the Cloud Shell:
gcloud iam service-accounts create striim-spanner --display-name striim-spanner export sa_striim_spanner=$(gcloud iam service-accounts list --filter="displayName:striim-spanner" --format='value(email)') export PROJECT=$(gcloud info --format='value(config.project)') gcloud projects add-iam-policy-binding $PROJECT --role roles/spanner.databaseUser --member serviceAccount:$sa_striim_spanner gcloud iam service-accounts keys create ~/striim-spanner-key.json --iam-account $sa_striim_spanner

This will create a key called striim-spanner-key.json in your home path.

Now you need to move the key you just created to the Striim VM.

  1. Identify the name of the virtual machine in which Striim is deployed on by either going to the deployment details page in the Deployment Manager, or by using gcloud compute instances list command in the Cloud Console. Now that you have the name of the Striim VM, continue with the instructions below.

  2. Copy the JSON file to the Striim server with the following command:

gcloud compute scp ~/striim-spanner-key.json $STRIIMVM_NAME:~ --zone=$STRIIMVM_ZONE

It might be that this causes an SSH key to be generated (in this case you have to provide a passphrase).

  1. Confirm when promoted to continue with Y and leave the passphrase field empty.
Note: There might be a workgroup error displayed in the console - you can safely ignore that.
  1. Now, move the JSON file on the Striim VM to the /opt/striim directory using this command:
gcloud compute ssh --zone=$STRIIMVM_ZONE $STRIIMVM_NAME -- 'sudo cp ~/striim-spanner-key.json /opt/striim && sudo chown striim /opt/striim/striim-spanner-key.json'

You are now ready to create a Striim application!

Click Check my progress to verify the objective. Create a Cloud Spanner access key

Task 4. Create an online database migration

An online database migration moves data from a source database (either on-premises or hosted on a cloud provider) to a target database in Google Cloud. The source database remains fully accessible by the business application and with minimal performance impact on the source database during this time.

In an online migration, an initial bulk load is performed, and a continuous capture of changes is also being run. You then synchronize the two databases to ensure that data isn't lost. Typically both databases will be retained for long periods of time to test and verify that the application and users are not impacted by switching to a new cloud database.

Start by creating an initial bulk load.

  1. In the Striim Web UI, click Apps pane.

  2. Click the Create App button on the upper right hand side of the page.

There are multiple ways to create applications in Striim.

  1. Click Start From Scratch.

  2. Name the application something that is easily remembered. This lab will use demo_online.

  3. Below the application name, you can choose a namespace. These are logical groupings that help you organize applications. Use the default admin namespace. Click Save.

You'll be greeted by the Flow Designer page. All of the out of the box connectors that you need to create streaming data applications are in the left hand menu bar.

  1. To do a one time bulk initial load of data, use the Database Reader.

  2. Drag and drop the Database tile from the Sources pane to the application canvas in the middle.

The Striim application canvas, with the New Database tile in the middle.

  1. Enter all of your connection properties on the right New Source.
Property Value (type value or select option as specified)
Name mysql_source
Connection URL jdbc:mysql://[PRIMARY_ADDRESS]:3306/striimdemo
Username striim-user
Password striim-usr-password
Note: [PRIMARY_ADDRESS] is the IP address of SQL instance. The IP address can be found through Cloud Shell with the command gcloud sql instances list.
  1. Click the Show Advanced Settings link to see more configuration properties.
Property Value (type value or select option as specified)
Tables striimdemo.ORDERS
Output to > New output stream_CloudSQLMySQLInitLoad
Note: Make a note of the settings in this step and the previous step as you will need it again in the next section.
  1. Click Save.

Now do a quick test to see if the configuration settings are correct, and confirm that Striim can successfully connect to MySQL.

  1. In the middle of the page there is a button marked Created with a drop down arrow next to it. Click the button, and select Deploy App.

In the Deployment window, this is where you specify that you want to run certain pieces of your application on certain parts of your deployment topology. You'll typically use this if you're running a lightweight forwarding agent on-premises deployed next to your source, pushing data over specific ports to the Striim server running in the cloud.

  1. Since you're using a simple deployment topology, select the default option and click Deploy.

  2. Click the wave icon beneath your Database Reader component, and then click the eye icon next to it to preview your data as it flows through the Striim pipeline.

The eye icon highlighted on the UI

  1. Go back up to the top menu bar, click Deployed, and then click Start App.

You should now see the Striim application running, and data flowing through the pipeline. If there are any errors, it means that there is an issue connecting to the source database since there is only a source component in the pipeline. If you see your application successfully run but no data flows through, typically that means that you don't have any existing data in your database.

The Application Progress page, which includes information such as the total output, input rate, and previous data streams.

You've successfully connected to your source database and tested that it can read data.

  1. Click Running on the top menu bar, and then select Stop App.

  2. Click Stopped, and then select Undeploy App. You are now ready to connect to Cloud Spanner.

  3. Click the wave button under the DatabaseReader. This time, a + button will appear. Click it, and then select Connect next Target component.

  4. Name this target object CloudSpannerInitialLoadTarget, and then search for "Spanner" under the Adapter field. Select SpannerWriter.

Spanner writer selected in the Adapter category.

  1. Enter the rest of the connection properties.
Property Value (type value or select option as specified)
Service Account Key /opt/striim/striim-spanner-key.json
InstanceID striim-spanner-demo
Tables striimdemo.ORDERS,striimdemo.orders
Note: You can always use gcloud spanner instances list in Cloud Shell to check. Note: The tables property is a source/target pair separated by commas. It will look like srcSchema1.srcTable1,tgtSchema1.tgtTable1; srcSchema2.srcTable2,tgtSchema2.tgtTable2. Note: The Service Account Key property is looking for a fully qualified path and name of the key file that was generated earlier.
  1. Click Save.

  2. Click Deploy App from the Created drop down.

  3. Select the default option and click Deploy.

  4. Click Deployed, and then click Start App.

  5. In the Cloud Console, navigate to Spanner, and then to your striimdemo database.

  6. Click Spanner Studio on the left pane.

  7. In the Editor window, run the query SELECT * from orders and the replicated data should return.

Note: It might take a minute for the data to appear due to the default configuration settings.

The Results tabbed page, which lists several orders and their details.

You have successfully set up your Striim environment and pipeline to perform a batch load.

  1. Since you're working with Cloud Spanner, run the following analytical query to verify that your data arrived properly:
SELECT COUNT(*) AS ORDERS, AVG(ORDER_TOTAL) AS ORDERS_AVE, SUM(ORDER_TOTAL) AS ORDERS_SUM FROM orders;

This query will find the average and total size of the orders. They should be 43148.952 and 431489.52 respectively.

The Results tabbed page, which lists one result.

Task 5. Create a continuous Cloud SQL for MySQL to Cloud Spanner data pipeline

With an initial one time bulk load in place, you still set up a continuous replication pipeline. It will look very similar to the bulk pipeline that you just created, but with a different source object.

  1. Click the Striim logo in the upper left of the screen to bring up the main menu. Click on Home to bring you back to the main page, click Apps, and then click the Create App button in the upper right corner to create a new app.

The Create App button dislayed on the UI.

  1. Click Start from Scratch, and then name it MySQLToCloudSpanner_cdc. Leave the namespace as it is: admin.

  2. Click Save.

  3. This time, instead of choosing a DatabaseReader source, choose a MySQL CDC reader.

The highlighted MySQL CDC tile in the Sources category.

  1. Enter all of your connection properties on the right New Source.
Property Value (type value or select option as specified)
Name spanner_source
Connection URL jdbc:mysql://[PRIMARY_ADDRESS]:3306/striimdemo
Username striim-user
Password striim-usr-password
Tables striimdemo.ORDERS
Output to > New output tgt_MySQLCDCSpanner
Note: [PRIMARY_ADDRESS] is the IP address of SQL instance. The IP address can be found through Cloud Shell with the command gcloud sql instances list.
  1. Click Save.

  2. Click the wave button under the Spanner_Source. This time, a + button will appear. Click it, and then select Connect next Target component.

  3. Name this target object tgt_CDC_SPANNER and then search for "Spanner" under the Adapter field. Select SpannerWriter.

  4. Enter the rest of the connection properties.

Property Value (type value or select option as specified)
Service Account Key /opt/striim/striim-spanner-key.json
InstanceID striim-spanner-demo
Tables striimdemo.ORDERS,striimdemo.orders
  1. Click Save.

  2. Now, click Deploy App from the Created drop down.

  3. Select the default option and click Deploy.

  4. Click Deployed, and then click Start App.

  5. Click the wave icon, and then the eye icon next to it to look at the data.

  6. Go back to Cloud Shell and run the following command to connect your MySQL instance:

gcloud sql connect striim-sql-src --user=striim-user
  1. It will prompt you for a password, enter striim-usr-password.

  2. In the striimdemo database, execute the following commands to add some more data:

USE striimdemo; INSERT INTO ORDERS (ORDER_ID, ORDER_DATE, ORDER_MODE, CUSTOMER_ID, ORDER_STATUS, ORDER_TOTAL, SALES_REP_ID, PROMOTION_ID) VALUES (1011, 1568928576017, 'In-Store', 1011, 9, 13879.56, 320, 88252); INSERT INTO ORDERS (ORDER_ID, ORDER_DATE, ORDER_MODE, CUSTOMER_ID, ORDER_STATUS, ORDER_TOTAL, SALES_REP_ID, PROMOTION_ID) VALUES (1012, 1568928636017, 'CompanyA', 1012, 1, 19729.99, 76, 95203); INSERT INTO ORDERS (ORDER_ID, ORDER_DATE, ORDER_MODE, CUSTOMER_ID, ORDER_STATUS, ORDER_TOTAL, SALES_REP_ID, PROMOTION_ID) VALUES (1013, 1568928696017, 'In-Store', 1013, 5, 7286.68, 164, 45162); INSERT INTO ORDERS (ORDER_ID, ORDER_DATE, ORDER_MODE, CUSTOMER_ID, ORDER_STATUS, ORDER_TOTAL, SALES_REP_ID, PROMOTION_ID) VALUES (1014, 1568928756017, 'Online', 1014, 1, 87268.61, 909, 70407); INSERT INTO ORDERS (ORDER_ID, ORDER_DATE, ORDER_MODE, CUSTOMER_ID, ORDER_STATUS, ORDER_TOTAL, SALES_REP_ID, PROMOTION_ID) VALUES (1015, 1568928816017, 'CompanyB', 1015, 1, 69744.13, 424, 79401);
  1. Go back to the Striim console. You should start to see the transactions flow through!
Note: It might take a few minutes for the transactions to start appearing due to the default configuration settings.

The Flow page, which displays the status: Running.

  1. Take a look at the table in Cloud Spanner by either querying it with the gcloud tool in Cloud Shell, or by navigating to the table in the Cloud Console, and you will see that the data has successfully replicated to the target.

The Data tabbed page, which lists several orders and their details.

  1. In the Cloud Console, navigate to Cloud Spanner, and then to your striimdemo dataset.

  2. In Spanner Studio, run one final query to verify that your data arrived properly. This finds the average order size as well as the total order sum. It should be 41959.899333333335 and 629398.4900000001 after the CDC pipeline, respectively.

SELECT COUNT(*) AS ORDERS, AVG(ORDER_TOTAL) AS ORDERS_AVE, SUM(ORDER_TOTAL) AS ORDERS_SUM FROM orders;

The Results tabbed page, with one search result populated.

Congratulations!

You have successfully set up a streaming pipeline from Cloud SQL for MySQL to Cloud Spanner using Striim, Google Cloud's Data Migration Partner.

Next steps / Learn more

Google Cloud training and certification

...helps you make the most of Google Cloud technologies. Our classes include technical skills and best practices to help you get up to speed quickly and continue your learning journey. We offer fundamental to advanced level training, with on-demand, live, and virtual options to suit your busy schedule. Certifications help you validate and prove your skill and expertise in Google Cloud technologies.

Last Tested Date December 5, 2023

Last Updated Date December 5, 2023

Copyright 2024 Google LLC All rights reserved. Google and the Google logo are trademarks of Google LLC. All other company and product names may be trademarks of the respective companies with which they are associated.