AX99 / shopping_insider

shopping markup reboot

Shopping Insider

Disclaimer: This is not an officially supported Google product.

Shopping Insider is a tool to enable retailers grow their business using Google Merchant Center by taking actionable data-driven decisions to optimize shopping feed health and ads performance.

Contents

• 1. Overview
  • 1.1. Value Proposition
  • 1.2. Solution Architecture
  • 1.3. Solution Options
• 2. Installation
  • 2.1. Environment Setup
  • 2.2. Installation Options
  • 2.3. Multiple Multi-Client Account(MCA) support

1. Overview

The Shopping Insider solution is built for Shopping Ads customers to take actionable data-driven decisions to improve their feed health and shopping ads performance.

1.1. Value Proposition

• Users can find opportunities and issues at each stage of the Shopping Funnel both overall and detailed data cuts.

• Richer insights with data joins to provide overall and product level performance information pivoted towards custom attributes (product type, brand, etc) for deeper insights.

• A dashboard to share data and insights across different teams and areas of the business seamlessly to address issues & optimize performance.

1.2 Solution Architecture

The solution will export data from GMC and Google Ads to your Google Cloud Project on a daily basis and provide insights via Looker Studio dashboard.

1.3 Solution Options

Please join this Google Group to gain the viewer access for some of the resources below. (i.e. templates, spreadsheets)

Shopping Insider

This is the base solution that exclusively uses the products and product issues tables available via the Merchant Center Transfer. This will allow you to set up the Shopping Insider Dashboard Template.

Shopping Insider + Market Insights [Work In Progress]

Stay tune for the updates!

2. Installation

2.1. Environment Setup

2.1.1 Create a GCP project with billing account

You may skip this step if you already have a GCP account with billing enabled.

• How to Create a GCP account (if you don't have one already!)

• How to Create and Manage Projects

• How to Create, Modify, or Close Your Billing Account

2.1.2 Check the permissions

Make sure the user running the installation has following permissions.

• Standard Access For GMC

• Standard Access For Google Ads

• Editor(or Owner) Role in Google Cloud Project

2.2. Installation Options

There are two ways you can install Shopping Insider:

Option 1: Install via Cyborg (Google Sheet), if you are not comfortable with Command Line Interface (CLI) and want a clear view of any failed component install, if any.

• Pros: Fast and easy to deploy. Displays what happens when Shopping Insider was installed and keeps a record of all kinds of GCP resources that were enabled, checked, created or updated. It’s easy to upgrade to a new version, e.g. the sql files are always downloaded from GitHub for the latest version. When there are optional features, Cyborg offers a way to easily reconfigure the features of the solution.

• Cons: The user who makes the copy of Cyborg should belong to the same org which owns the GCP or have a Google Account, as they need to be able to change AppScript project number.

Option 2: Install via Shell script, if you are comfortable with Command Line Interface (CLI) and need more details into what is getting installed beforehand.

• Pros: will support any client GCP structure

• Cons: longer and more technical to deploy than via Cyborg

2.2.1. Option 1: Install via Cyborg(Google Sheet)

2.2.1.1. Make a copy of the tool

1. Join the Google Group group and wait for approval.
2. After you join the group, you can visit the Google Sheets tool and make a copy.

2.2.1.2. Configure the OAuth consent screen

If there is no OAuth consent screen in this GCP project, you need to configure the OAuth consent screen first. When you create the consent screen, some settings need to be:

1. Publishing status as In production, otherwise the refresh token will expire every 7 days.
2. User type could be External or check User type for more details.
3. No scopes need to be filled in.

2.2.1.3. Update GCP project number to your Google Sheet

1. Get your GCP project number. See how to determine the project number of a standard Cloud project.
2. Click Google Sheets menu Extensions -> Apps Script to open Apps Script editor window.
3. On the Apps Script window, click ⚙️ (Project Settings) at the left menu bar, then click the button Change project.
4. Enter the project number and click the button Set project.

2.2.1.4. Deploy Shopping Insider

To install Shopping Insider:

1. Switch to sheet Shopping Insider and input required information in the sheet, including GMC Account Id, Google Ads MCC and Project Id.
2. Click menu 🤖 Cyborg -> Shopping Insider -> Check resources to run a check. If an error happened, fix it and retry Check resources.
3. If there are resources marked as TO_APPLY, use menu 🤖 Cyborg -> Shopping Insider -> Apply changes to apply the modifications.
4. If there are resources not checked, continue to step 2.
5. After all resources are marked as OK, click the Dashboard Template link to make a copy. You need to confirm and save the dashboard in the opened window.

Note: When you first time click the menu item, an OAuth authorization window may prompt you to grant permissions. After you complete it, you need to click the menu item again to continue.

Note: Some processes, e.g. waiting for a newly created Data Transfer to finish the first run takes time. If there was a timeout, then wait sometime and come back retry Check Resources.

Note: Why TO_APPLY? Some operations required user inputs, for example, the location of a new BigQuery dataset. Cyborg will pause there and ask you to select a location and click menu Apply Changes as a confirmation.

2.2.2. Option 2: Install via Shell Script(command line)

2.2.2.1. Setup local environment.

Download and authenticate gcloud.

Alternatively, if the GMC account has less than 50 Million products, you could use Cloud Shell, which comes with gcloud already installed. The cloud shell disconnects after 1 hour and hence we recommend using local environment for large accounts since they could take more than 1 hour to finish the installation.

2.2.2.2. Check out source codes

Open the cloud shell or your terminal(if running locally) and clone the repository.

  git clone https://github.com/google/shopping_insider

2.2.2.3 Run install script

Please provide following inputs when running the setup.sh script:

• GCP Project Id

• Google Merchant Center Id

• Google Ads External Customer Id

cd shopping-insider;
sh setup.sh --project_id=<project_id> --merchant_id=<merchant_id> --ads_customer_id=<ads_customer_id>

When installing, the script will check whether the current user has the proper authorization to continue. It may ask you to open cloud authorization URL in the browser. Please follow the instructions as mentioned in the command line.

Note - If the script fails when you run it for the first time, it might be due to delay in preparing Merchant account data. Usually accounts with massive data set. Please wait up to 1-3 days before re-running the script.

During the installation process, the script will do following:

• Enable Google Cloud Components and Google APIs

  • BigQuery

  • BigQuery Data Transfer

• Create Google Merchant Center and Google Ads data transfers.

• Create recurring data transfer jobs so that the latest data is imported in near real time.

• Create following Shopping Insider specific SQL tables.

  • product_detailed_materialized - Latest snapshot view of products combined with performance metrics. Each offer is split into rows for each targeted country, rows are keyed by unique_product_id and target_country.
  • product_historical_materialized - Historic snapshot of performance metrics at a product category level.

2.2.2.4. [Optional] Update location and locales if different than US

• If your data shouldn't be materialized in US, change the BigQuery dataset location in config.yaml

• [Market Insights only] Adjust the locales in best_sellers_workflow.sql, by default set to "en-US"

• You could make the changes before running the install script or after

  • If you're making the changes afterwards, re-run the install script
  • Check the scheduled queries in BigQuery and disable any older version of the Main Workflow

2.2.2.5. Configure Data Sources

You will need to create or copy required Data Source(s) in Data Studio:

For Shopping Insider:

• Create product_detailed_materialized Data Source (linked to shopping_insider.product_detailed_materialized)
• Create product_historical_materialized Data Source (linked to shopping_insider.product_historical_materialized)

To create a data source:

• Click on the link

• Make sure you are using BigQuery connector. If not choose "BigQuery" from the list of available connectors.

• Search your GCP Project Id under My Projects.

• Under Dataset, click on "shopping_insider".

• Under Table, choose the required table view.

• Click Connect on the top right corner and wait for the data-source to be created

To copy a data source:

• Click on the data source template link above.

• Click on the icon in the top right corner next to "Create Report".

• Click "Copy Data Source" on the "Copy Data Source" pop-up.

• Select your Project, Dataset, and Table to be connected, then press "Reconnect" in the top right corner.

• Click "Apply" on the "Apply Connection Changes" pop-up

• Repeat this process for all three data source templates above.

2.2.2.6. Create Data-Studio Dashboard(s)

For Shopping Insider:

• Click on the following link to the Looker Studio template: link

• Click "Use my own data"

• Replace data sources by choosing the new "product_detailed_materialized" and "product_historical_materialized" data-sources created in the previous step

• Click "Edit and share"

Note - The performance metrics in the dashboard might take 12-24 hours to appear.

2.3. Multiple Multi-Client Account(MCA) support

1. If you have more than one Google Merchant Center, repeat the installation steps for all MCA.

2. Creates a data set in Big Query (Guide).

3. Creates the views to union all the data set from step 1.

   CREATE OR REPLACE VIEW `<project_id>.<final_dataset>.product_detailed_materialized`
   AS (
     SELECT
       *
     FROM
       `<project_id>.<dataset_1>.product_detailed_materialized`
     UNION ALL
     SELECT
       *
     FROM
       `<project_id>.<dataset_2>.product_detailed_materialized`
     ......
   );
   

   CREATE OR REPLACE VIEW `<project_id>.<final_dataset>.product_historical_materialized`
   AS (
     SELECT
       *
     FROM
       `<project_id>.<dataset_1>.product_historical_materialized`
     UNION ALL
     SELECT
       *
     FROM
       `<project_id>.<dataset_2>.product_historical_materialized`
     ......
   );
   

4. Replaces the dashboard data sources with the views.