Skip to content

Weather Raw Function

get_grid(connection, parameters_dict)

A function to return back raw data by querying databricks SQL Warehouse using a connection specified by the user.

This will return the raw weather forecast data for a grid.

The available connectors by RTDIP are Databricks SQL Connect, PYODBC SQL Connect, TURBODBC SQL Connect.

The available authentication methods are Certificate Authentication, Client Secret Authentication or Default Authentication. See documentation.

This function requires the user to input a dictionary of parameters which are focused on Weather Data. (See Attributes table below)

Parameters:

Name Type Description Default
connection object

Connection chosen by the user (Databricks SQL Connect, PYODBC SQL Connect, TURBODBC SQL Connect)

 required
parameters_dict dict

A dictionary of parameters (see Attributes table below)

 required

Attributes:

Name Type Description
source optional str

Source of the data the full table name
forecast str

Any specific identifier for forecast
forecast_type(str) str

Type of forecast ie weather, solar, power, etc
region str

Region
data_security_level str

Level of data security
data_type str

Type of the data (float, integer, double, string)
start_date str

Start date (Either a date in the format YY-MM-DD or a datetime in the format YYY-MM-DDTHH:MM:SS or specify the timezone offset in the format YYYY-MM-DDTHH:MM:SS+zz:zz)
end_date str

End date (Either a date in the format YY-MM-DD or a datetime in the format YYY-MM-DDTHH:MM:SS or specify the timezone offset in the format YYYY-MM-DDTHH:MM:SS+zz:zz)
forecast_run_start_date str

Start date of the forecast run (Either a date in the format YY-MM-DD or a datetime in the format YYY-MM-DDTHH:MM:SS or specify the timezone offset in the format YYYY-MM-DDTHH:MM:SS+zz:zz)
forecast_run_end_date str

End date of the forecast run (Either a date in the format YY-MM-DD or a datetime in the format YYY-MM-DDTHH:MM:SS or specify the timezone offset in the format YYYY-MM-DDTHH:MM:SS+zz:zz)
timestamp_column str

The column which contains the the forecast output time. Default "EventTime".
forecast_run_timestamp_column str

The column which contains whent the forecast was run. Default "EnqueuedTime".
max_lat float

Maximum latitude
max_lon float

Maximum longitude
min_lat float

Minimum latitude
min_lon float

Minimum longitude
measurement optional str

Measurement type
limit optional int

The number of rows to be returned

}

Returns:

Name Type Description
DataFrame DataFrame

A dataframe of raw weather forecast data.
Source code in src/sdk/python/rtdip_sdk/queries/weather/raw.py 
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
def get_grid(connection: object, parameters_dict: dict) -> pd.DataFrame:
    """
    A function to return back raw data by querying databricks SQL Warehouse using a connection specified by the user.

    This will return the raw weather forecast data for a grid.

    The available connectors by RTDIP are Databricks SQL Connect, PYODBC SQL Connect, TURBODBC SQL Connect.

    The available authentication methods are Certificate Authentication, Client Secret Authentication or Default Authentication. See documentation.

    This function requires the user to input a dictionary of parameters which are focused on Weather Data. (See Attributes table below)

    Args:
        connection: Connection chosen by the user (Databricks SQL Connect, PYODBC SQL Connect, TURBODBC SQL Connect)
        parameters_dict: A dictionary of parameters (see Attributes table below)

    Attributes:
        source (optional str): Source of the data the full table name
        forecast (str): Any specific identifier for forecast
        forecast_type(str): Type of forecast ie weather, solar, power, etc
        region (str): Region
        data_security_level (str): Level of data security
        data_type (str): Type of the data (float, integer, double, string)
        start_date (str): Start date (Either a date in the format YY-MM-DD or a datetime in the format YYY-MM-DDTHH:MM:SS or specify the timezone offset in the format YYYY-MM-DDTHH:MM:SS+zz:zz)
        end_date (str): End date (Either a date in the format YY-MM-DD or a datetime in the format YYY-MM-DDTHH:MM:SS or specify the timezone offset in the format YYYY-MM-DDTHH:MM:SS+zz:zz)
        forecast_run_start_date (str): Start date of the forecast run (Either a date in the format YY-MM-DD or a datetime in the format YYY-MM-DDTHH:MM:SS or specify the timezone offset in the format YYYY-MM-DDTHH:MM:SS+zz:zz)
        forecast_run_end_date (str): End date of the forecast run (Either a date in the format YY-MM-DD or a datetime in the format YYY-MM-DDTHH:MM:SS or specify the timezone offset in the format YYYY-MM-DDTHH:MM:SS+zz:zz)
        timestamp_column (str): The column which contains the the forecast output time. Default "EventTime".
        forecast_run_timestamp_column (str): The column which contains whent the forecast was run. Default "EnqueuedTime".
        max_lat (float): Maximum latitude
        max_lon (float): Maximum longitude
        min_lat (float): Minimum latitude
        min_lon (float): Minimum longitude
        measurement (optional str): Measurement type
        limit (optional int): The number of rows to be returned
    }

    Returns:
        DataFrame: A dataframe of raw weather forecast data.
    """
    try:
        query = _query_builder(parameters_dict, "raw_grid")

        print(query)

        try:
            cursor = connection.cursor()
            cursor.execute(query)
            df = cursor.fetch_all()
            cursor.close()
            connection.close()
            return df
        except Exception as e:
            logging.exception("error returning dataframe")
            raise e

    except Exception as e:
        logging.exception("error with raw function")
        raise e

get_point(connection, parameters_dict)

A function to return back raw data by querying databricks SQL Warehouse using a connection specified by the user.

This will return the raw weather forecast data for a single point.

The available connectors by RTDIP are Databricks SQL Connect, PYODBC SQL Connect, TURBODBC SQL Connect.

The available authentication methods are Certificate Authentication, Client Secret Authentication or Default Authentication. See documentation.

This function requires the user to input a dictionary of parameters which are focused on Weather Data. (See Attributes table below)

Parameters:

Name Type Description Default
connection object

Connection chosen by the user (Databricks SQL Connect, PYODBC SQL Connect, TURBODBC SQL Connect)

 required
parameters_dict dict

A dictionary of parameters (see Attributes table below)

 required

Attributes:

Name Type Description
source optional str

Source of the data the full table name
forecast str

Any specific identifier for forecast
forecast_type(str) str

Type of forecast ie weather, solar, power, etc
region str

Region
data_security_level str

Level of data security
data_type str

Type of the data (float, integer, double, string)
start_date str

Start date (Either a date in the format YY-MM-DD or a datetime in the format YYY-MM-DDTHH:MM:SS or specify the timezone offset in the format YYYY-MM-DDTHH:MM:SS+zz:zz)
end_date str

End date (Either a date in the format YY-MM-DD or a datetime in the format YYY-MM-DDTHH:MM:SS or specify the timezone offset in the format YYYY-MM-DDTHH:MM:SS+zz:zz)
forecast_run_start_date str

Start date of the forecast run (Either a date in the format YY-MM-DD or a datetime in the format YYY-MM-DDTHH:MM:SS or specify the timezone offset in the format YYYY-MM-DDTHH:MM:SS+zz:zz)
forecast_run_end_date str

End date of the forecast run (Either a date in the format YY-MM-DD or a datetime in the format YYY-MM-DDTHH:MM:SS or specify the timezone offset in the format YYYY-MM-DDTHH:MM:SS+zz:zz)
timestamp_column str

The column which contains the the forecast output time. Default "EventTime".
forecast_run_timestamp_column str

The column which contains whent the forecast was run. Default "EnqueuedTime.
lat float

latitude
lon float

longitude
measurement optional str

Measurement type
limit optional int

The number of rows to be returned

}

Returns:

Name Type Description
DataFrame DataFrame

A dataframe of raw weather forecast data.
Source code in src/sdk/python/rtdip_sdk/queries/weather/raw.py 
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
def get_point(connection: object, parameters_dict: dict) -> pd.DataFrame:
    """
    A function to return back raw data by querying databricks SQL Warehouse using a connection specified by the user.

    This will return the raw weather forecast data for a single point.

    The available connectors by RTDIP are Databricks SQL Connect, PYODBC SQL Connect, TURBODBC SQL Connect.

    The available authentication methods are Certificate Authentication, Client Secret Authentication or Default Authentication. See documentation.

    This function requires the user to input a dictionary of parameters which are focused on Weather Data. (See Attributes table below)

    Args:
        connection: Connection chosen by the user (Databricks SQL Connect, PYODBC SQL Connect, TURBODBC SQL Connect)
        parameters_dict: A dictionary of parameters (see Attributes table below)

    Attributes:
        source (optional str): Source of the data the full table name
        forecast (str): Any specific identifier for forecast
        forecast_type(str): Type of forecast ie weather, solar, power, etc
        region (str): Region
        data_security_level (str): Level of data security
        data_type (str): Type of the data (float, integer, double, string)
        start_date (str): Start date (Either a date in the format YY-MM-DD or a datetime in the format YYY-MM-DDTHH:MM:SS or specify the timezone offset in the format YYYY-MM-DDTHH:MM:SS+zz:zz)
        end_date (str): End date (Either a date in the format YY-MM-DD or a datetime in the format YYY-MM-DDTHH:MM:SS or specify the timezone offset in the format YYYY-MM-DDTHH:MM:SS+zz:zz)
        forecast_run_start_date (str): Start date of the forecast run (Either a date in the format YY-MM-DD or a datetime in the format YYY-MM-DDTHH:MM:SS or specify the timezone offset in the format YYYY-MM-DDTHH:MM:SS+zz:zz)
        forecast_run_end_date (str): End date of the forecast run (Either a date in the format YY-MM-DD or a datetime in the format YYY-MM-DDTHH:MM:SS or specify the timezone offset in the format YYYY-MM-DDTHH:MM:SS+zz:zz)
        timestamp_column (str): The column which contains the the forecast output time. Default "EventTime".
        forecast_run_timestamp_column (str): The column which contains whent the forecast was run. Default "EnqueuedTime.
        lat (float): latitude
        lon (float): longitude
        measurement (optional str): Measurement type
        limit (optional int): The number of rows to be returned
    }

    Returns:
        DataFrame: A dataframe of raw weather forecast data.
    """
    try:
        query = _query_builder(parameters_dict, "raw_point")

        print(query)

        try:
            cursor = connection.cursor()
            cursor.execute(query)
            df = cursor.fetch_all()
            cursor.close()
            connection.close()
            return df
        except Exception as e:
            logging.exception("error returning dataframe")
            raise e

    except Exception as e:
        logging.exception("error with raw function")
        raise e

Example get_point

from rtdip_sdk.authentication.azure import DefaultAuth
from rtdip_sdk.queries.weather.raw import get_point
from rtdip_sdk.connectors import DatabricksSQLConnection

auth = DefaultAuth().authenticate()
token = auth.get_token("2ff814a6-3304-4ab8-85cb-cd0e6f879c1d/.default").token
connection = DatabricksSQLConnection("{server_hostname}", "{http_path}", token)

params = {
    "forecast": "mock_forecast",
    "forecast_type": "mock_weather",
    "region": "mock_region",
    "data_security_level": "mock_security",
    "data_type": "mock_data_type",
    "lat": 1.1,
    "lon": 1.1,
    "start_date": "2020-01-01",
    "end_date": "2020-01-02",
    "forecast_run_start_date": "2020-01-01",
    "forecast_run_end_date": "2020-01-02",
    "timestamp_column": "EventTime",
    "forecast_run_timestamp_column": "EnqueuedTime",
}

x = get_point(connection, params)

print(x)

Example get_grid

from rtdip_sdk.authentication.azure import DefaultAuth
from rtdip_sdk.queries.weather.raw import get_grid
from rtdip_sdk.connectors import DatabricksSQLConnection

auth = DefaultAuth().authenticate()
token = auth.get_token("2ff814a6-3304-4ab8-85cb-cd0e6f879c1d/.default").token
connection = DatabricksSQLConnection("{server_hostname}", "{http_path}", token)

params = {
    "forecast": "mock_forecast",
    "forecast_type": "mock_weather",
    "region": "mock_region",
    "data_security_level": "mock_security",
    "data_type": "mock_data_type",
    "min_lat": 36,
    "max_lat": 38,
    "min_lon": -109.1,
    "max_lon": -107.1,
    "start_date": "2020-01-01",
    "end_date": "2020-01-02",
    "forecast_run_start_date": "2020-01-01",
    "forecast_run_end_date": "2020-01-02",
    "timestamp_column": "EventTime",
    "forecast_run_timestamp_column": "EnqueuedTime",
}

x = get_grid(connection, params)

print(x)

These examples are using DefaultAuth() and DatabricksSQLConnection() to authenticate and connect. You can find other ways to authenticate here. The alternative built in connection methods are either by PYODBCSQLConnection(), TURBODBCSQLConnection() or SparkConnection().

Note

server_hostname and http_path can be found on the SQL Warehouses Page.