Control timestamp¶

BigQuery Doesn't Store Timezones Alongside Timestamps

bookmark_border Problem User loads data with time values into BigQuery. These values are converted to timestamps and stored, however, the information about the timezone they came from is lost. Environment BigQuery dataset with loaded time data. Solution Storing timezone information is not best practice. The application should store all timestamps as a single TZ (ideally UTC) and present them as needed.

If the user still wants to preserve the information about the timezone, or any additional information related to time data, they need to store it separately, for example, in a new column of their dataset.

Cause The timestamps are being stored as long integers (basically the timestamp is a number of milliseconds passed since 01 January 1970 (UTC), so it does not depend on timezone). When user loads a timestamp to BigQuery, it is automatically converted to a long integer using timezone provided (or assuming UTC if no timezone provided).

https://cloud.google.com/knowledge/kb/bigquery-doesn-t-store-timezones-alongside-timestamps-000004468

The origin of flow component¶

s1[Source Asia/HoChiMInh] -- convert timestamp UTC --> lake[Datalake using only UTC] -- parse back --> target

Python control¶

Create the timestamp to UTC

Using native

tzdata==2023.3

datetime.now(tz=ZoneInfo("Asia/Ho_Chi_Minh")).locate(ZoneInfo("UTC"))

At the lake, it can be load back and transfer to timestampe

at the element componenbt