Raw Function

`get(connection, parameters_dict)`

A function to return back raw data by querying databricks SQL Warehouse using a connection specified by the user.

The available connectors by RTDIP are Databricks SQL Connect, PYODBC SQL Connect, TURBODBC SQL Connect.

The available authentication methods are Certificate Authentication, Client Secret Authentication or Default Authentication. See documentation.

This function requires the user to input a dictionary of parameters. (See Attributes table below)

Parameters:

Name	Type	Description	Default
`connection`	`object`	Connection chosen by the user (Databricks SQL Connect, PYODBC SQL Connect, TURBODBC SQL Connect)	required
`parameters_dict`	`dict`	A dictionary of parameters (see Attributes table below)	required

Attributes:

Name	Type	Description
`business_unit`	`str`	Business unit
`region`	`str`	Region
`asset`	`str`	Asset
`data_security_level`	`str`	Level of data security
`data_type`	`str`	Type of the data (float, integer, double, string)
`tag_names`	`list`	List of tagname or tagnames ["tag_1", "tag_2"]
`start_date`	`str`	Start date (Either a date in the format YY-MM-DD or a datetime in the format YYY-MM-DDTHH:MM:SS or specify the timezone offset in the format YYYY-MM-DDTHH:MM:SS+zz:zz)
`end_date`	`str`	End date (Either a date in the format YY-MM-DD or a datetime in the format YYY-MM-DDTHH:MM:SS or specify the timezone offset in the format YYYY-MM-DDTHH:MM:SS+zz:zz)
`include_bad_data`	`bool`	Include "Bad" data points with True or remove "Bad" data points with False
`limit`	`optional int`	The number of rows to be returned
`offset`	`optional int`	The number of rows to skip before returning rows
`case_insensitivity_tag_search`	`optional bool`	Search for tags using case insensitivity with True or case sensitivity with False

Returns:

Name	Type	Description
`DataFrame`	`DataFrame`	A dataframe of raw timeseries data.

Warning

Setting case_insensitivity_tag_search to True will result in a longer query time.

Source code in src/sdk/python/rtdip_sdk/queries/time_series/raw.py

def get(connection: object, parameters_dict: dict) -> pd.DataFrame:
    """
    A function to return back raw data by querying databricks SQL Warehouse using a connection specified by the user.

    The available connectors by RTDIP are Databricks SQL Connect, PYODBC SQL Connect, TURBODBC SQL Connect.

    The available authentication methods are Certificate Authentication, Client Secret Authentication or Default Authentication. See documentation.

    This function requires the user to input a dictionary of parameters. (See Attributes table below)

    Args:
        connection: Connection chosen by the user (Databricks SQL Connect, PYODBC SQL Connect, TURBODBC SQL Connect)
        parameters_dict: A dictionary of parameters (see Attributes table below)

    Attributes:
        business_unit (str): Business unit
        region (str): Region
        asset (str): Asset
        data_security_level (str): Level of data security
        data_type (str): Type of the data (float, integer, double, string)
        tag_names (list): List of tagname or tagnames ["tag_1", "tag_2"]
        start_date (str): Start date (Either a date in the format YY-MM-DD or a datetime in the format YYY-MM-DDTHH:MM:SS or specify the timezone offset in the format YYYY-MM-DDTHH:MM:SS+zz:zz)
        end_date (str): End date (Either a date in the format YY-MM-DD or a datetime in the format YYY-MM-DDTHH:MM:SS or specify the timezone offset in the format YYYY-MM-DDTHH:MM:SS+zz:zz)
        include_bad_data (bool): Include "Bad" data points with True or remove "Bad" data points with False
        limit (optional int): The number of rows to be returned
        offset (optional int): The number of rows to skip before returning rows
        case_insensitivity_tag_search (optional bool): Search for tags using case insensitivity with True or case sensitivity with False

    Returns:
        DataFrame: A dataframe of raw timeseries data.

    !!! warning
        Setting `case_insensitivity_tag_search` to True will result in a longer query time.
    """
    try:
        query = _query_builder(parameters_dict, "raw")

        try:
            cursor = connection.cursor()
            cursor.execute(query)
            df = cursor.fetch_all()
            cursor.close()
            connection.close()
            return df
        except Exception as e:
            logging.exception("error returning dataframe")
            raise e

    except Exception as e:
        logging.exception("error with raw function")
        raise e

Example

from rtdip_sdk.authentication.azure import DefaultAuth
from rtdip_sdk.connectors import DatabricksSQLConnection
from rtdip_sdk.queries import raw

auth = DefaultAuth().authenticate()
token = auth.get_token("2ff814a6-3304-4ab8-85cb-cd0e6f879c1d/.default").token
connection = DatabricksSQLConnection("{server_hostname}", "{http_path}", token)

parameters = {
    "business_unit": "{business_unit}",
    "region": "{region}", 
    "asset": "{asset_name}", 
    "data_security_level": "{security_level}", 
    "data_type": "float",
    "tag_names": ["{tag_name_1}", "{tag_name_2}"],
    "start_date": "2023-01-01",
    "end_date": "2023-01-31",
    "include_bad_data": True,
}
x = raw.get(connection, parameters)
print(x)

This example is using DefaultAuth() and DatabricksSQLConnection() to authenticate and connect. You can find other ways to authenticate here. The alternative built in connection methods are either by PYODBCSQLConnection(), TURBODBCSQLConnection() or SparkConnection().

Note

See Samples Repository for full list of examples.

Note

server_hostname and http_path can be found on the SQL Warehouses Page.