Exporting Salesforce CRM data into BigQuery www.e-nor.com | 1

Deeper Salesforce and GA360 Insights with

BigQuery and Google CloudBy Mahmoud Taha

www.e-nor.com

Exporting Salesforce CRM data into BigQuery www.e-nor.com | 2

About the AuthorMahmoud TahaData Engineering Team Lead

Mahmoud is all about facing challenges, the harder the challenge the sweeter the victory. After receiving his Bachelor degree in Computer Science and a diploma in Software Development he worked as a software developer. Later switching careers to the Business Intelligence and Data Analysis field in which he found his passion working with data, analysis and revealing hidden insights to help companies make better business decisions. Mahmoud is kind and funny and loves spending time with his family, traveling and hitting the gym.

Exporting Salesforce CRM data into BigQuery www.e-nor.com | 3

Companies can benefit from integrating CRM data with other data sources to build one powerful data warehouse. Once done, the possibilites are endless, but first you need to have the right tools and sources to build such data warehouse. With BigQuery, as one of the most powerful serverless data warehouses in the market, we can benefit from including Salesforce Sales Cloud data in BigQuery and then using it with Ads Data Hub or joining wth other tools that have native exports/data transfer services such as Google Analytics 360, Firebase, or Campaign Manager. While this post is specifically addressing exporting Salesforce data, the methodology described here applies to most CRM systems or any other platform that offers an API for extracting data.

Why export Salesforce data into BigQuery?

Salesforce and Google Analytics 360 already offer critical integrations, so why do we still want to export Salesforce data into BigQuery (alongside the automated export to BigQuery that is routinely activated for Google Analytics 360)?

The high-level answer is that the size, speed, and scalability of BigQuery opens a whole lot of possiblities for integration, advanced analysis, and optimization based on your Salesforce data on its own, and even more possibilities when you’re also exporting joinable

Salesforce Sales Cloud and Marketing Cloud Integration with Google Analytics 360

Exporting your Salesforce and Google Analytics 360 data to BigQuery will offer many additional opportunities for insight and action, but make sure to also take advantage of the exciting capabilities of the direct integration between Salesforce and Google Analytics 360.

The benefits of the Sales and Marketing Cloud integrations with GA 360 include:• viewing offline Opportunity stage and Lead status changes as Google Analytics

events• report won Opportunities as Google Analytics ecommerce transactions,• model attribution for the original traffic sources that contributed to the offline

conversion• create Custom Funnels based on offline and online activity to identify drop off

points• remarket based on online and offline activity, also taking advantage of Similar

Audiences in Display & Video 360 and Google Ads to target additional prospects• segment your email lists based on online activity, such as completion step 4 of a

5-step conversion process viewing 50% of your company video

For more about this integration, sign up for the Google Solution Guide Library to download the Analytics 360 Salesforce Cloud Solutions Guide and other helpful guides.

Exporting Salesforce CRM data into BigQuery www.e-nor.com | 4

Of the many Salesforce APIs available, we’ll be using the REST API for this integration.

Google Analytics 360 data to BigQuery.

The specific answer for you depends on your needs. What do you need to understand about customers’ behavior? Is there a specific product or discount most new acquisitions go for? Why is the retention rate higher in some locations than others? and so on.

Here are some use cases:• Visualization: BigQuery integrates with several powerful dashboarding tools for

advanced analytics and richer insights. By joining Salesforce data with GA data in BigQuery, you can tie up events or goal completion from GA with leads and opportunities from Salesforce.

• Targeted marketing: analyze the market to understand customer trends, predict needs, and optimize promotional efforts.

• Understand customers’ behavior: using CRM with sales data, you can monitor how their products are performing online and accordingly improve their sales or marketing strategies.

• Higher retention rates: by understanding and predicting potential abandonments or customer inactivity, you can provide better retention offers in a more timely manner.

• Advanced modeling: using CRM data, you can build various models - like customer segmentation, loyalty and inactivity - for better targeting.

• Preserving data: exporting to BigQuery will help save historical data for later analysis. And avoid losing any fields that are updated or deleted from Salesforce.

How to import Salesforce data into BigQuery?Salesforce has a bunch of APIs for developers. Choosing the right API for integration is an important decision. Here is a list of the available APIs.

API Name Protocol Data Format CommunicationREST API REST JSON, XML Synchronous

SOAP API SOAP (WSDL) XML Synchronous

Chatter REST API REST JSON, XML Synchronous (photos are processed asynchronously)

User interface API REST JSON Synchronous

Analytics API REST JSON, XML Synchronous

Bulk API REST CSV, JSON, XML Asynchronous

Metadata API SOAP (WSDL) XML Asynchronous

Streaming API Bayeux JSON Asynchronous (stream of data)

Apex REST API REST JSON, XML, Custom SynchronousApex SOAP API SOAP (WSDL) XML Synchronous

Tooling API REST or SOAP (WSDL) JSON, XML, Custom Synchronous

Exporting Salesforce CRM data into BigQuery www.e-nor.com | 5

1. Set up Salesforce for authentication In this example we’ll be using the REST API, but before connecting to Salesforce through an API we need to perform some steps for authentication as follows:

• Create a new “Connected App” in the Salesforce portal. Go in “Setup”, open the “App Manager”. Then, create a “New Connected App”.

• Fill in the mandatory fields (Name and Email). Then, tick the box “Enable OAuth Settings” and In “Selected OAuth Scopes”, make all scopes available. Finally, Type “http://localhost/” in “Callback URL”.

• After the app is created, you’ll get a “Customer Key” and “Customer Secret” which will be used later for authentication.

• You’ll also need to have the “Security token”. In case you don’t know it you’ll need to reset it. Go to Settings --> Reset My Security Token --> Reset Security Token as shown below, then you’ll receive the new security token via email. Copy and save.

Creating a Connected App in App Manager

In case you don’t know your security token for accessing the Salesforce REST API, you’ll need to reset your security token.

Exporting Salesforce CRM data into BigQuery www.e-nor.com | 6

2. Use Python for building the pipelineIn our case we’ll be building this pipeline using Google Cloud Platform tools:

• Cloud Functions• Cloud Scheduler• Cloud Storage

First, we’ll create a HTTP triggered Cloud Function to make the call to Salesforce REST API; you can write the function in Python (beta), Node.JS or Go (beta). We’ll be using Python.

Cloud Function for connecting to the Salesforce REST API and saving to Cloud Storage,triggered by a HTTP request from Cloud Scheduler

Exporting Salesforce CRM data into BigQuery www.e-nor.com | 7

The code snippet below is the body of the Cloud Function. It will post an authentication request to get back an Access Token which will be used to authorize the APIs requests. 3. Call the APIAfter the function retrieves the Access Token and the Instance URL we can now start making calls to the Salesforce REST API and extract our data. In the example below we’ll extract all leads. As we said, the goal here is to link Salesforce data with GA or any other data source that is exported to BigQuery, so we need to have a common key to join on. In our case we have the GA client Id exported to Salesforce from GA, if you want to learn more about how to do this step please refer to this post. First, we construct the header and use the access token for authorization, then perform our API call, specifying the:

• method “GET”• action (this link lists all available actions supported by REST APIs)• params (including the query to execute)

import requestsparams = { “grant_type”: “password”, “client_id”: “XXX.YYY”, # Consumer Key “client_secret”: “*****”, # Consumer Secret “username”: “Login Email Address”, # The email used to login “password”: “PasswordSecurityToken” # Concat password and security token}r = requests.post(“https://login.salesforce.com/services/oauth2/token”, params=params)access_token = r.json().get(“access_token”)instance_url = r.json().get(“instance_url”)

Authentication request for an access token

header = { ‘Content-type’: ‘application/json’, ‘Accept-Encoding’: ‘gzip’, ‘Authorization’: ‘Bearer {}’.format(access_token) }

r = requests.request(‘get’, instance_url+’/services/data/v39.0/query/’, headers=header, params={‘q’: ‘Select GACID__c, Company, FirstName, LastName, LeadSource,Status From Lead’}, timeout=30) # Where GACID__c is the GA Client ID

Constructing the header for authorization and then requesting the data

Exporting Salesforce CRM data into BigQuery www.e-nor.com | 8

4. Extract the response dataOnce the request is successful we can simply extract the response JSON object. Then, the most efficient way to load data into BigQuery is to first write the data to a json file in Cloud Storage bucket, then load that file to a BigQuery table.

The below code extracts the JSON response and writes it to a .json file on Cloud Storage.

5. Load data into BigQueryNext step is to read the data from the JSON file into BigQuery. Create a Cloud Function with “Cloud Storage” trigger, and “Finalize/Create” as event type, to fire once a file is uploaded into a Cloud Storage bucket.

import jsonjsonString = json.dumps(r.json(), indent=2)

bucket_name = ‘MyBucket’ #Bucket name on Cloud Storageblob_name = ‘DatasetName+TableName.json’ # The file to be created

from google.cloud import storage

storage_client = storage.Client()bucket = storage_client.get_bucket(bucket_name)blob = bucket.blob(blob_name)

blob.upload_from_string(jsonString)

Extracting the response JSON and writing to Cloud Storage

A second Cloud Function for populating a BigQuery table, triggered by the JSON file saved in Cloud Storage.

Exporting Salesforce CRM data into BigQuery www.e-nor.com | 9

The function will parse the file name to retrieve the data set name and the table name to be created or append the data into. This is performed by the below code snippet.

def jsontobq(data, context):

GCP_PROJECT = os.environ.get(‘GCP_PROJECT’) # get storage update data bucketname = data[‘bucket’] filename = data[‘name’]

# Validate filename dataset_name+table_name.json if not re.search(‘^[A-Za-z_]+\+[A-Za-z_]+.json$’, filename): logging.error(‘Invalid file name please use this formate Dataset+TableName.json’) return

# parse filename datasetname, tablename = filename.replace(‘.json’, ‘’).split(‘+’) table_id = ‘%s.%s.%s’ % (GCP_PROJECT, datasetname, tablename)

# construct GS uri uri = ‘gs://%s/%s’ % (bucketname, filename)

# create bigquery client client = bigquery.Client()

# get dataset reference dataset_ref = client.dataset(datasetname) # create a bigquery load job config job_config = bigquery.LoadJobConfig() job_config.autodetect = True job_config.create_disposition = ‘CREATE_IF_NEEDED’ job_config.source_format = ‘NEWLINE_DELIMITED_JSON’ job_config.write_disposition = ‘WRITE_APPEND’

# create a bigquery load job try: load_job = client.load_table_from_uri( uri,table_id,job_config=job_config) except Exception as e: logging.error(‘Failed to load file to BQ: %s’ % (e))

Body of Cloud Function parses the JSON file name for the BigQuery data set and table name and then reads the Salesforce data into BigQuery accordingly.

Exporting Salesforce CRM data into BigQuery www.e-nor.com | 10

6. Scheduling the daily exportTo schedule the daily export, we’ll use Cloud Scheduler to kick start the process on a daily basis by calling the HTTP triggered Cloud Function. To set up a Cloud Scheduler, from the UI go to Cloud Scheduler and click “Create job” as below. Then, enter the scheduler details, for the “Target” select “HTTP”, and in the “URL” enter the URL of the HTTP triggered Cloud Function. Cloud Scheduler provides four types of HTTP requests:

• POST: used to send data to a server. The data sent to the server with POST is stored in the request body of the HTTP request; it’ll be our choice in this use case although we won’t send any data in the body.

• GET: used to request data from a specified resource. This is helpful in use cases where data needs to be retrieved from a server, in which case the response of the HTTP GET request will be logged in a “scheduler.logging.AttemptFinished” event.

• PUT: similar to POST requests, but PUT requests are idempotent, meaning calling the same PUT request multiple times will always produce the same result.

• DELETE: deletes the specified resource from a server.

Frequency field can be a little tricky. ts format is “* * * * *” and can be configured down to the minutes level, the five asterisks represent “minute 0-59, hour 0-23, day of month 1-31, month 1-12 and day of week 0-7”, below are some frequency examples:

• “*/45 * * * *” fires every 45 mins• “0 9 * * 1” every Monday at 09:00• “0 */8 * * *” every 8 hours• “* 6 2 * *” at 06:00 on the 2nd of each month

Exporting Salesforce CRM data into BigQuery www.e-nor.com | 11

Cloud Scheduler will kickstart the entire process by making an HTTP request to trigger the Cloud Function that writes Salesforce data to BigQuery.

Exporting Salesforce CRM data into BigQuery www.e-nor.com | 12

Finally, hit create and the scheduler should be up and ready for next run. You can also run it manually for testing or debugging.

The below diagram summarizes the flow of the export process.

ConclusionImporting Salesforce’s CRM data into BigQuery using the power of GCP tools is a fairly easy process, and can be done with just few configurations and some code snippets. But, easy as it is, it opens a lot of possibilities as well for advanced analysis, data modeling and even maching learning or predictive modeling. Especially when you have all the horsepower of BigQuery at your fingertips.

To learn more about how E-Nor can help you take full advantage of Salesforce, Google Analytics, Campaign Manager, and other enterprise data sets for advanced analysis on Google Cloud Platform, contact us today.