RevenueCat -> Cloud Storage -> BigQuery, Data Transfer Configuration

2 years ago
11 March 2022
5 replies
358 views

Stepan Parunashvili
New Member
0 replies

Hey team, I have the integration that sends a daily dump of our subscription info into Cloud Storage.

Now, I’d like to create a daily job that imports this into BigQuery. I look at the “Data Transfer” Configuration, and am a bit stuck with “Cloud Storage URI”:

How could I get it to _only_ get the CSV for a certain date string.

I know this is more of a Google Cloud question, but I’d guess everyone who exports to Storage wants to do this too, so thought I’d ask here: if you could share your data transfer configuration with me would appreciate it!

5 replies

@Stepan Parunashvili I am running into the same issue! Did you ever find a way to get the data into BQ?

Alright, I think I figured it out (mostly)! Turns out that BQ allows wildcards when creating a table from a GCS folder, so I was able to point BQ to the base folder (the one that holds all of the data folders) and to reference that folder with a wildcard. The resulting table had the file from every date.

What is still missing with this approach is the actual date from the date file. All of the csvs are joined together with no way of determining which date the particular record existed in.