You can catalog, document, and query live data from BigQuery by configuring a virtual connection to it. With a virtual connection, your data stays where it is--it is not hosted on data.world--so all actions using it are always run against current data.
When connecting your data systems to data.world, often your system or firewall may have a network policy that only allows access from specific IP addresses--your allowlist. In those cases, you must add data.world to your network policy IP allowlist.
Allow the following IP addresses:
52.3.83.134
52.205.195.10
52.205.207.86
If you are on your organization's home page select Connection Manager. Otherwise go to the integration page (https://data.world/integrations/bigquery) and click Enable integration to set up a connection.
When you create a connection to use with a data source you are asked to set the owner of the connection. If you are in organization you have the choice of making your organization the owner of the connection or making it a personal connection.
There are two compelling reasons for making most connections owned by the organization:
There is no loss of access to data when an employee leaves and their account is deactivated.
Federation across data sources is faster and more efficient if it uses the same connection.
Organizational-level connections are shared between admins of the organization and can be used by by all of them to create new live tables. Non-administrator users can only query and preview existing live tables.
Note
Organization-owned connections can only be used to add data to datasets owned by that organization. If you are in organizations A and B, you cannot add data to a dataset owned by B using a connection owned by A.
With a personal connection, only the connection owner may create new live tables with the connection, and other members of the organization can query and preview live tables.
In order to create a virtual connection to BigQuery you will use the following parameters:
Project ID: The unique identifier for your BigQuery project
Service account username: A Google account that is associated with your Google project, as opposed to a specific user
Service account key file: Provides the authentication information used in the connection configuration. This file will be uploaded when you enter the other configuration information into the dialog.
Choose a Connection owner (yourself or your organization) and set the Display name for your connection. The display name is the name everyone in your organization will see for the connection. Then enter your user credential information into the dialog screen:
Click Test BigQuery configuration to make sure it works, and then save it by selecting Configure.
After your connection is configured you can use it anytime you select Add data:
Choose the connection you want to use from My data sources:
Go to the integration page (easily found under My integrations on our Integrations page) and select Manage:
From here you can edit your current connection or add a new one:
Note
You will need your original credentials (password or key file) to make changes to an existing connection.
When executing queries against virtualized data sources in data.world, we will translate those queries to the proper SQL dialect of the target system and run them on the target system whenever possible.
When functions cannot be translated directly, data.world fetch the necessary data from the target system and execute those functions locally. We refer to those as emulated functions or aggregations below.
Additionally, some functions may not be supported on either the target system or locally - those are noted as unavailable functions below.
Emulated functions
regex (only 3-argument version emulated)
regexp_extract (only 3-argument version emulated)
Emulated aggregations
regr_avgx (only DISTINCT emulated)
regr_avgy (only DISTINCT emulated)
regr_count (only DISTINCT emulated)
regr_intercept (only DISTINCT emulated)
regr_r2 (only DISTINCT emulated)
regr_slope (only DISTINCT emulated)
regr_sxx (only DISTINCT emulated)
regr_sxy (only DISTINCT emulated)
regr_syy (only DISTINCT emulated)
Unavailable functions
Database connectors - A full list of our connection integrations