arkouda.io ========== .. py:module:: arkouda.io Functions --------- .. autoapisummary:: arkouda.io.export arkouda.io.get_columns arkouda.io.get_datasets arkouda.io.get_filetype arkouda.io.get_null_indices arkouda.io.import_data arkouda.io.load arkouda.io.load_all arkouda.io.ls arkouda.io.ls_csv arkouda.io.read arkouda.io.read_csv arkouda.io.read_hdf arkouda.io.read_parquet arkouda.io.read_tagged_data arkouda.io.read_zarr arkouda.io.receive arkouda.io.receive_dataframe arkouda.io.restore arkouda.io.save_all arkouda.io.snapshot arkouda.io.to_csv arkouda.io.to_hdf arkouda.io.to_parquet arkouda.io.to_zarr arkouda.io.update_hdf Module Contents --------------- .. py:function:: export(read_path: str, dataset_name: str = 'ak_data', write_file: Optional[str] = None, return_obj: bool = True, index: bool = False) Export data from Arkouda file (Parquet/HDF5) to Pandas object or file formatted to be readable by Pandas :param read_path: path to file where arkouda data is stored. :type read_path: str :param dataset_name: name to store dataset under :type dataset_name: str :param index: Default False. When True, maintain the indexes loaded from the pandas file :type index: bool :param write_file: path to file to write pandas formatted data to. Only write the file if this is set :type write_file: str, optional :param return_obj: Default True. When True return the Pandas DataFrame object, otherwise return None :type return_obj: bool, optional :raises RuntimeError: - Unsupported file type :returns: When `return_obj=True` :rtype: pd.DataFrame .. seealso:: :obj:`pandas.DataFrame.to_parquet`, :obj:`pandas.DataFrame.to_hdf`, :obj:`pandas.DataFrame.read_parquet`, :obj:`pandas.DataFrame.read_hdf`, :obj:`ak.import_data` .. rubric:: Notes - If Arkouda file is exported for pandas, the format will not change. This mean parquet files will remain parquet and hdf5 will remain hdf5. - Export can only be performed from hdf5 or parquet files written by Arkouda. The result will be the same file type, but formatted to be read by Pandas. .. py:function:: get_columns(filenames: Union[str, List[str]], col_delim: str = ',', allow_errors: bool = False) -> List[str] Get a list of column names from CSV file(s). .. py:function:: get_datasets(filenames: Union[str, List[str]], allow_errors: bool = False, column_delim: str = ',', read_nested: bool = True) -> List[str] Get the names of the datasets in the provide files :param filenames: Name of the file/s from which to return datasets :type filenames: str or List[str] :param allow_errors: Default: False Whether or not to allow errors while accessing datasets :type allow_errors: bool :param column_delim: Column delimiter to be used if dataset is CSV. Otherwise, unused. :type column_delim: str :param read_nested: Default True, when True, SegArray objects will be read from the file. When False, SegArray (or other nested Parquet columns) will be ignored. Only used for Parquet Files. :type read_nested: bool :rtype: List[str] of names of the datasets :raises RuntimeError: - If no datasets are returned .. rubric:: Notes - This function currently supports HDF5 and Parquet formats. - Future updates to Parquet will deprecate this functionality on that format, but similar support will be added for Parquet at that time. - If a list of files is provided, only the datasets in the first file will be returned .. seealso:: :obj:`ls` .. py:function:: get_filetype(filenames: Union[str, List[str]]) -> str Get the type of a file accessible to the server. Supported file types and possible return strings are 'HDF5' and 'Parquet'. :param filenames: A file or list of files visible to the arkouda server :type filenames: Union[str, List[str]] :returns: Type of the file returned as a string, either 'HDF5', 'Parquet' or 'CSV :rtype: str :raises ValueError: Raised if filename is empty or contains only whitespace .. rubric:: Notes - When list provided, it is assumed that all files are the same type - CSV Files without the Arkouda Header are not supported .. seealso:: :obj:`read_parquet`, :obj:`read_hdf` .. py:function:: get_null_indices(filenames: Union[str, List[str]], datasets: Optional[Union[str, List[str]]] = None) -> Union[arkouda.pdarrayclass.pdarray, Mapping[str, arkouda.pdarrayclass.pdarray]] Get null indices of a string column in a Parquet file. :param filenames: Either a list of filenames or shell expression :type filenames: list or str :param datasets: (List of) name(s) of dataset(s) to read. Each dataset must be a string column. There is no default value for this function, the datasets to be read must be specified. :type datasets: list or str or None :returns: Dictionary of {datasetName: pdarray} :rtype: returns a dictionary of Arkouda pdarrays :raises RuntimeError: Raised if one or more of the specified files cannot be opened. :raises TypeError: Raised if we receive an unknown arkouda_type returned from the server .. seealso:: :obj:`get_datasets`, :obj:`ls` .. py:function:: import_data(read_path: str, write_file: Optional[str] = None, return_obj: bool = True, index: bool = False) Import data from a file saved by Pandas (HDF5/Parquet) to Arkouda object and/or a file formatted to be read by Arkouda. :param read_path: path to file where pandas data is stored. This can be glob expression for parquet formats. :type read_path: str :param write_file: path to file to write arkouda formatted data to. Only write file if provided :type write_file: str, optional :param return_obj: Default True. When True return the Arkouda DataFrame object, otherwise return None :type return_obj: bool, optional :param index: Default False. When True, maintain the indexes loaded from the pandas file :type index: bool, optional :raises RuntimeWarning: - Export attempted on Parquet file. Arkouda formatted Parquet files are readable by pandas. :raises RuntimeError: - Unsupported file type :returns: When `return_obj=True` :rtype: pd.DataFrame .. seealso:: :obj:`pandas.DataFrame.to_parquet`, :obj:`pandas.DataFrame.to_hdf`, :obj:`pandas.DataFrame.read_parquet`, :obj:`pandas.DataFrame.read_hdf`, :obj:`ak.export` .. rubric:: Notes - Import can only be performed from hdf5 or parquet files written by pandas. .. py:function:: load(path_prefix: str, file_format: str = 'INFER', dataset: str = 'array', calc_string_offsets: bool = False, column_delim: str = ',') -> Union[Mapping[str, Union[arkouda.pdarrayclass.pdarray, arkouda.strings.Strings, arkouda.segarray.SegArray, arkouda.categorical.Categorical, arkouda.dataframe.DataFrame, arkouda.client_dtypes.IPv4, arkouda.timeclass.Datetime, arkouda.timeclass.Timedelta, arkouda.index.Index]]] Load a pdarray previously saved with ``pdarray.save()``. :param path_prefix: Filename prefix used to save the original pdarray :type path_prefix: str :param file_format: 'INFER', 'HDF5' or 'Parquet'. Defaults to 'INFER'. Used to indicate the file type being loaded. If INFER, this will be detected during processing :type file_format: str :param dataset: Dataset name where the pdarray was saved, defaults to 'array' :type dataset: str :param calc_string_offsets: If True the server will ignore Segmented Strings 'offsets' array and derive it from the null-byte terminators. Defaults to False currently :type calc_string_offsets: bool :param column_delim: Column delimiter to be used if dataset is CSV. Otherwise, unused. :type column_delim: str :returns: Dictionary of {datsetName: Union[pdarray, Strings, SegArray, Categorical]} with the previously saved pdarrays, Strings, SegArrays, or Categoricals :rtype: Mapping[str, Union[pdarray, Strings, SegArray, Categorical]] :raises TypeError: Raised if either path_prefix or dataset is not a str :raises ValueError: Raised if invalid file_format or if the dataset is not present in all hdf5 files or if the path_prefix does not correspond to files accessible to Arkouda :raises RuntimeError: Raised if the hdf5 files are present but there is an error in opening one or more of them .. seealso:: :obj:`to_parquet`, :obj:`to_hdf`, :obj:`load_all`, :obj:`read` .. rubric:: Notes If you have a previously saved Parquet file that is raising a FileNotFound error, try loading it with a .parquet appended to the prefix_path. Parquet files were previously ALWAYS stored with a ``.parquet`` extension. ak.load does not support loading a single file. For loading single HDF5 files without the _LOCALE#### suffix please use ak.read(). CSV files without the Arkouda Header are not supported. .. rubric:: Examples >>> # Loading from file without extension >>> obj = ak.load('path/prefix') Loads the array from numLocales files with the name ``cwd/path/name_prefix_LOCALE####``. The file type is inferred during processing. >>> # Loading with an extension (HDF5) >>> obj = ak.load('path/prefix.test') Loads the object from numLocales files with the name ``cwd/path/name_prefix_LOCALE####.test`` where #### is replaced by each locale numbers. Because filetype is inferred during processing, the extension is not required to be a specific format. .. py:function:: load_all(path_prefix: str, file_format: str = 'INFER', column_delim: str = ',', read_nested=True) -> Mapping[str, Union[arkouda.pdarrayclass.pdarray, arkouda.strings.Strings, arkouda.segarray.SegArray, arkouda.categorical.Categorical]] Load multiple pdarrays, Strings, SegArrays, or Categoricals previously saved with ``save_all()``. :param path_prefix: Filename prefix used to save the original pdarray :type path_prefix: str :param file_format: 'INFER', 'HDF5', 'Parquet', or 'CSV'. Defaults to 'INFER'. Indicates the format being loaded. When 'INFER' the processing will detect the format Defaults to 'INFER' :type file_format: str :param column_delim: Column delimiter to be used if dataset is CSV. Otherwise, unused. :type column_delim: str :param read_nested: Default True, when True, SegArray objects will be read from the file. When False, SegArray (or other nested Parquet columns) will be ignored. Parquet files only :type read_nested: bool :returns: Dictionary of {datsetName: Union[pdarray, Strings, SegArray, Categorical]} with the previously saved pdarrays, Strings, SegArrays, or Categoricals :rtype: Mapping[str, Union[pdarray, Strings, SegArray, Categorical]] :raises TypeError:: Raised if path_prefix is not a str :raises ValueError: Raised if file_format/extension is encountered that is not hdf5 or parquet or if all datasets are not present in all hdf5/parquet files or if the path_prefix does not correspond to files accessible to Arkouda :raises RuntimeError: Raised if the hdf5 files are present but there is an error in opening one or more of them .. seealso:: :obj:`to_parquet`, :obj:`to_hdf`, :obj:`load`, :obj:`read` .. rubric:: Notes This function has been updated to determine the file extension based on the file format variable This function will be deprecated when glob flags are added to read_* methods CSV files without the Arkouda Header are not supported. .. py:function:: ls(filename: str, col_delim: str = ',', read_nested: bool = True) -> List[str] This function calls the h5ls utility on a HDF5 file visible to the arkouda server or calls a function that imitates the result of h5ls on a Parquet file. :param filename: The name of the file to pass to the server :type filename: str :param col_delim: The delimiter used to separate columns if the file is a csv :type col_delim: str :param read_nested: Default True, when True, SegArray objects will be read from the file. When False, SegArray (or other nested Parquet columns) will be ignored. Only used for Parquet files. :type read_nested: bool :returns: The string output of the datasets from the server :rtype: str :raises TypeError: Raised if filename is not a str :raises ValueError: Raised if filename is empty or contains only whitespace :raises RuntimeError: Raised if error occurs in executing ls on an HDF5 file :raises Notes: - This will need to be updated because Parquet will not technically support this when we update. Similar functionality will be added for Parquet in the future - For CSV files without headers, please use ls_csv .. seealso:: :obj:`ls_csv` .. py:function:: ls_csv(filename: str, col_delim: str = ',') -> List[str] Used for identifying the datasets within a file when a CSV does not have a header. :param filename: The name of the file to pass to the server :type filename: str :param col_delim: The delimiter used to separate columns if the file is a csv :type col_delim: str :returns: The string output of the datasets from the server :rtype: str .. seealso:: :obj:`ls` .. py:function:: read(filenames: Union[str, List[str]], datasets: Optional[Union[str, List[str]]] = None, iterative: bool = False, strictTypes: bool = True, allow_errors: bool = False, calc_string_offsets=False, column_delim: str = ',', read_nested: bool = True, has_non_float_nulls: bool = False, fixed_len: int = -1) -> Union[Mapping[str, Union[arkouda.pdarrayclass.pdarray, arkouda.strings.Strings, arkouda.segarray.SegArray, arkouda.categorical.Categorical, arkouda.dataframe.DataFrame, arkouda.client_dtypes.IPv4, arkouda.timeclass.Datetime, arkouda.timeclass.Timedelta, arkouda.index.Index]]] Read datasets from files. File Type is determined automatically. :param filenames: Either a list of filenames or shell expression :type filenames: list or str :param datasets: (List of) name(s) of dataset(s) to read (default: all available) :type datasets: list or str or None :param iterative: Iterative (True) or Single (False) function call(s) to server :type iterative: bool :param strictTypes: If True (default), require all dtypes of a given dataset to have the same precision and sign. If False, allow dtypes of different precision and sign across different files. For example, if one file contains a uint32 dataset and another contains an int64 dataset with the same name, the contents of both will be read into an int64 pdarray. :type strictTypes: bool :param allow_errors: Default False, if True will allow files with read errors to be skipped instead of failing. A warning will be included in the return containing the total number of files skipped due to failure and up to 10 filenames. :type allow_errors: bool :param calc_string_offsets: Default False, if True this will tell the server to calculate the offsets/segments array on the server versus loading them from HDF5 files. In the future this option may be set to True as the default. :type calc_string_offsets: bool :param column_delim: Column delimiter to be used if dataset is CSV. Otherwise, unused. :type column_delim: str :param read_nested: Default True, when True, SegArray objects will be read from the file. When False, SegArray (or other nested Parquet columns) will be ignored. Ignored if datasets is not None Parquet Files only. :type read_nested: bool :param has_non_float_nulls: Default False. This flag must be set to True to read non-float parquet columns that contain null values. :type has_non_float_nulls: bool :param fixed_len: Default -1. This value can be set for reading Parquet string columns when the length of each string is known at runtime. This can allow for skipping byte calculation, which can have an impact on performance. :type fixed_len: int :returns: Dictionary of {datasetName: pdarray, String, or SegArray} :rtype: Returns a dictionary of Arkouda pdarrays, Arkouda Strings, or Arkouda Segarrays. :raises RuntimeError: If invalid filetype is detected .. seealso:: :obj:`get_datasets`, :obj:`ls`, :obj:`read_parquet`, :obj:`read_hdf` .. rubric:: Notes If filenames is a string, it is interpreted as a shell expression (a single filename is a valid expression, so it will work) and is expanded with glob to read all matching files. If iterative == True each dataset name and file names are passed to the server as independent sequential strings while if iterative == False all dataset names and file names are passed to the server in a single string. If datasets is None, infer the names of datasets from the first file and read all of them. Use ``get_datasets`` to show the names of datasets to HDF5/Parquet files. CSV files without the Arkouda Header are not supported. .. rubric:: Examples Read with file Extension >>> x = ak.read('path/name_prefix.h5') # load HDF5 - processing determines file type not extension Read without file Extension >>> x = ak.read('path/name_prefix.parquet') # load Parquet Read Glob Expression >>> x = ak.read('path/name_prefix*') # Reads HDF5 .. py:function:: read_csv(filenames: Union[str, List[str]], datasets: Optional[Union[str, List[str]]] = None, column_delim: str = ',', allow_errors: bool = False) -> Union[Mapping[str, Union[arkouda.pdarrayclass.pdarray, arkouda.strings.Strings, arkouda.segarray.SegArray, arkouda.categorical.Categorical, arkouda.dataframe.DataFrame, arkouda.client_dtypes.IPv4, arkouda.timeclass.Datetime, arkouda.timeclass.Timedelta, arkouda.index.Index]]] Read CSV file(s) into Arkouda objects. If more than one dataset is found, the objects will be returned in a dictionary mapping the dataset name to the Arkouda object containing the data. If the file contains the appropriately formatted header, typed data will be returned. Otherwise, all data will be returned as a Strings object. :param filenames: The filenames to read data from :type filenames: str or List[str] :param datasets: names of the datasets to read. When `None`, all datasets will be read. :type datasets: str or List[str] (Optional) :param column_delim: The delimiter for column names and data. Defaults to ",". :type column_delim: str :param allow_errors: Default False, if True will allow files with read errors to be skipped instead of failing. A warning will be included in the return containing the total number of files skipped due to failure and up to 10 filenames. :type allow_errors: bool :returns: Dictionary of {datasetName: pdarray, String, or SegArray} :rtype: Returns a dictionary of Arkouda pdarrays, Arkouda Strings, or Arkouda Segarrays. :raises ValueError: Raised if all datasets are not present in all parquet files or if one or more of the specified files do not exist :raises RuntimeError: Raised if one or more of the specified files cannot be opened. If `allow_errors` is true this may be raised if no values are returned from the server. :raises TypeError: Raised if we receive an unknown arkouda_type returned from the server .. seealso:: :obj:`to_csv` .. rubric:: Notes - CSV format is not currently supported by load/load_all operations - The column delimiter is expected to be the same for column names and data - Be sure that column delimiters are not found within your data. - All CSV files must delimit rows using newline (``\n``) at this time. - Unlike other file formats, CSV files store Strings as their UTF-8 format instead of storing bytes as uint(8). .. py:function:: read_hdf(filenames: Union[str, List[str]], datasets: Optional[Union[str, List[str]]] = None, iterative: bool = False, strict_types: bool = True, allow_errors: bool = False, calc_string_offsets: bool = False, tag_data=False) -> Union[Mapping[str, Union[arkouda.pdarrayclass.pdarray, arkouda.strings.Strings, arkouda.segarray.SegArray, arkouda.categorical.Categorical, arkouda.dataframe.DataFrame, arkouda.client_dtypes.IPv4, arkouda.timeclass.Datetime, arkouda.timeclass.Timedelta, arkouda.index.Index]]] Read Arkouda objects from HDF5 file/s :param filenames: Filename/s to read objects from :type filenames: str, List[str] :param datasets: datasets to read from the provided files :type datasets: Optional str, List[str] :param iterative: Iterative (True) or Single (False) function call(s) to server :type iterative: bool :param strict_types: If True (default), require all dtypes of a given dataset to have the same precision and sign. If False, allow dtypes of different precision and sign across different files. For example, if one file contains a uint32 dataset and another contains an int64 dataset with the same name, the contents of both will be read into an int64 pdarray. :type strict_types: bool :param allow_errors: Default False, if True will allow files with read errors to be skipped instead of failing. A warning will be included in the return containing the total number of files skipped due to failure and up to 10 filenames. :type allow_errors: bool :param calc_string_offsets: Default False, if True this will tell the server to calculate the offsets/segments array on the server versus loading them from HDF5 files. In the future this option may be set to True as the default. :type calc_string_offsets: bool :param tagData: Default False, if True tag the data with the code associated with the filename that the data was pulled from. :type tagData: bool :returns: Dictionary of {datasetName: pdarray, String, SegArray} :rtype: Returns a dictionary of Arkouda pdarrays, Arkouda Strings, or Arkouda Segarrays. :raises ValueError: Raised if all datasets are not present in all hdf5 files or if one or more of the specified files do not exist :raises RuntimeError: Raised if one or more of the specified files cannot be opened. If `allow_errors` is true this may be raised if no values are returned from the server. :raises TypeError: Raised if we receive an unknown arkouda_type returned from the server .. rubric:: Notes If filenames is a string, it is interpreted as a shell expression (a single filename is a valid expression, so it will work) and is expanded with glob to read all matching files. If iterative == True each dataset name and file names are passed to the server as independent sequential strings while if iterative == False all dataset names and file names are passed to the server in a single string. If datasets is None, infer the names of datasets from the first file and read all of them. Use ``get_datasets`` to show the names of datasets to HDF5 files. .. seealso:: :obj:`read_tagged_data` .. rubric:: Examples >>> # Read with file Extension >>> x = ak.read_hdf('path/name_prefix.h5') # load HDF5 # Read Glob Expression >>> x = ak.read_hdf('path/name_prefix*') # Reads HDF5 .. py:function:: read_parquet(filenames: Union[str, List[str]], datasets: Optional[Union[str, List[str]]] = None, iterative: bool = False, strict_types: bool = True, allow_errors: bool = False, tag_data: bool = False, read_nested: bool = True, has_non_float_nulls: bool = False, fixed_len: int = -1) -> Union[Mapping[str, Union[arkouda.pdarrayclass.pdarray, arkouda.strings.Strings, arkouda.segarray.SegArray, arkouda.categorical.Categorical, arkouda.dataframe.DataFrame, arkouda.client_dtypes.IPv4, arkouda.timeclass.Datetime, arkouda.timeclass.Timedelta, arkouda.index.Index]]] Read Arkouda objects from Parquet file/s :param filenames: Filename/s to read objects from :type filenames: str, List[str] :param datasets: datasets to read from the provided files :type datasets: Optional str, List[str] :param iterative: Iterative (True) or Single (False) function call(s) to server :type iterative: bool :param strict_types: If True (default), require all dtypes of a given dataset to have the same precision and sign. If False, allow dtypes of different precision and sign across different files. For example, if one file contains a uint32 dataset and another contains an int64 dataset with the same name, the contents of both will be read into an int64 pdarray. :type strict_types: bool :param allow_errors: Default False, if True will allow files with read errors to be skipped instead of failing. A warning will be included in the return containing the total number of files skipped due to failure and up to 10 filenames. :type allow_errors: bool :param tagData: Default False, if True tag the data with the code associated with the filename that the data was pulled from. :type tagData: bool :param read_nested: Default True, when True, SegArray objects will be read from the file. When False, SegArray (or other nested Parquet columns) will be ignored. If datasets is not None, this will be ignored. :type read_nested: bool :param has_non_float_nulls: Default False. This flag must be set to True to read non-float parquet columns that contain null values. :type has_non_float_nulls: bool :param fixed_len: Default -1. This value can be set for reading Parquet string columns when the length of each string is known at runtime. This can allow for skipping byte calculation, which can have an impact on performance. :type fixed_len: int :returns: Dictionary of {datasetName: pdarray, String, or SegArray} :rtype: Returns a dictionary of Arkouda pdarrays, Arkouda Strings, or Arkouda Segarrays. :raises ValueError: Raised if all datasets are not present in all parquet files or if one or more of the specified files do not exist :raises RuntimeError: Raised if one or more of the specified files cannot be opened. If `allow_errors` is true this may be raised if no values are returned from the server. :raises TypeError: Raised if we receive an unknown arkouda_type returned from the server .. rubric:: Notes If filenames is a string, it is interpreted as a shell expression (a single filename is a valid expression, so it will work) and is expanded with glob to read all matching files. If iterative == True each dataset name and file names are passed to the server as independent sequential strings while if iterative == False all dataset names and file names are passed to the server in a single string. If datasets is None, infer the names of datasets from the first file and read all of them. Use ``get_datasets`` to show the names of datasets to Parquet files. Parquet always recomputes offsets at this time This will need to be updated once parquets workflow is updated .. seealso:: :obj:`read_tagged_data` .. rubric:: Examples Read without file Extension >>> x = ak.read_parquet('path/name_prefix.parquet') # load Parquet Read Glob Expression >>> x = ak.read_parquet('path/name_prefix*') # Reads Parquet .. py:function:: read_tagged_data(filenames: Union[str, List[str]], datasets: Optional[Union[str, List[str]]] = None, strictTypes: bool = True, allow_errors: bool = False, calc_string_offsets=False, read_nested: bool = True, has_non_float_nulls: bool = False) Read datasets from files and tag each record to the file it was read from. File Type is determined automatically. :param filenames: Either a list of filenames or shell expression :type filenames: list or str :param datasets: (List of) name(s) of dataset(s) to read (default: all available) :type datasets: list or str or None :param strictTypes: If True (default), require all dtypes of a given dataset to have the same precision and sign. If False, allow dtypes of different precision and sign across different files. For example, if one file contains a uint32 dataset and another contains an int64 dataset with the same name, the contents of both will be read into an int64 pdarray. :type strictTypes: bool :param allow_errors: Default False, if True will allow files with read errors to be skipped instead of failing. A warning will be included in the return containing the total number of files skipped due to failure and up to 10 filenames. :type allow_errors: bool :param calc_string_offsets: Default False, if True this will tell the server to calculate the offsets/segments array on the server versus loading them from HDF5 files. In the future this option may be set to True as the default. :type calc_string_offsets: bool :param read_nested: Default True, when True, SegArray objects will be read from the file. When False, SegArray (or other nested Parquet columns) will be ignored. Ignored if datasets is not `None` Parquet Files only. :type read_nested: bool :param has_non_float_nulls: Default False. This flag must be set to True to read non-float parquet columns that contain null values. :type has_non_float_nulls: bool .. rubric:: Notes Not currently supported for Categorical or GroupBy datasets .. rubric:: Examples Read files and return data with tagging corresponding to the Categorical returned cat.codes will link the codes in data to the filename. Data will contain the code `Filename_Codes` >>> data, cat = ak.read_tagged_data('path/name') >>> data {'Filname_Codes': array([0 3 6 9 12]), 'col_name': array([0 0 0 1])} .. py:function:: read_zarr(store_path: str, ndim: int, dtype) Reads a Zarr store from disk into a pdarray. Supports multi-dimensional pdarrays of numeric types. To use this function, ensure you have installed the blosc dependency (`make install-blosc`) and have included `ZarrMsg.chpl` in the `ServerModules.cfg` file. :param store_path: The path to the Zarr store. The path must be to a directory that contains a `.zarray` file containing the Zarr store metadata. :type store_path: str :param ndim: The number of dimensions in the array :type ndim: int :param dtype: The data type of the array :type dtype: str :returns: The pdarray read from the Zarr store. :rtype: pdarray .. py:function:: receive(hostname: str, port) Receive a pdarray sent by `pdarray.transfer()`. :param hostname: The hostname of the pdarray that sent the array :type hostname: str :param port: The port to send the array over. This needs to be an open port (i.e., not one that the Arkouda server is running on). This will open up `numLocales` ports, each of which in succession, so will use ports of the range {port..(port+numLocales)} (e.g., running an Arkouda server of 4 nodes, port 1234 is passed as `port`, Arkouda will use ports 1234, 1235, 1236, and 1237 to send the array data). This port much match the port passed to the call to `pdarray.transfer()`. :type port: int_scalars :returns: The pdarray sent from the sending server to the current receiving server. :rtype: pdarray :raises ValueError: Raised if the op is not within the pdarray.BinOps set :raises TypeError: Raised if other is not a pdarray or the pdarray.dtype is not a supported dtype .. py:function:: receive_dataframe(hostname: str, port) Receive a pdarray sent by `dataframe.transfer()`. :param hostname: The hostname of the dataframe that sent the array :type hostname: str :param port: The port to send the dataframe over. This needs to be an open port (i.e., not one that the Arkouda server is running on). This will open up `numLocales` ports, each of which in succession, so will use ports of the range {port..(port+numLocales)} (e.g., running an Arkouda server of 4 nodes, port 1234 is passed as `port`, Arkouda will use ports 1234, 1235, 1236, and 1237 to send the array data). This port much match the port passed to the call to `pdarray.send_array()`. :type port: int_scalars :returns: The dataframe sent from the sending server to the current receiving server. :rtype: pdarray :raises ValueError: Raised if the op is not within the pdarray.BinOps set :raises TypeError: Raised if other is not a pdarray or the pdarray.dtype is not a supported dtype .. py:function:: restore(filename) Return data saved using `ak.snapshot` :param filename: :type filename: str :param Name used to create snapshot to be read: :rtype: Dict .. rubric:: Notes Unlike other save/load methods using snapshot restore will save DataFrames alongside other objects in HDF5. Thus, they are returned within the dictionary as a dataframe. .. py:function:: save_all(columns: Union[Mapping[str, Union[arkouda.pdarrayclass.pdarray, arkouda.strings.Strings, arkouda.segarray.SegArray]], List[Union[arkouda.pdarrayclass.pdarray, arkouda.strings.Strings, arkouda.segarray.SegArray]]], prefix_path: str, names: Optional[List[str]] = None, file_format='HDF5', mode: str = 'truncate', file_type: str = 'distribute', compression: Optional[str] = None) -> None DEPRECATED Save multiple named pdarrays to HDF5/Parquet files. :param columns: Collection of arrays to save :type columns: dict or list of pdarrays :param prefix_path: Directory and filename prefix for output files :type prefix_path: str :param names: Dataset names for the pdarrays :type names: list of str :param file_format: 'HDF5' or 'Parquet'. Defaults to hdf5 :type file_format: str :param mode: By default, truncate (overwrite) the output files if they exist. If 'append', attempt to create new dataset in existing files. :type mode: {'truncate' | 'append'} :param file_type: Default: distribute Single writes the dataset to a single file Distribute writes the dataset to a file per locale Only used with HDF5 :type file_type: str ("single" | "distribute") :param compression: Optional Select the compression to use with Parquet files. Only used with Parquet. :type compression: str (None | "snappy" | "gzip" | "brotli" | "zstd" | "lz4") :rtype: None :raises ValueError: Raised if (1) the lengths of columns and values differ or (2) the mode is not 'truncate' or 'append' .. seealso:: :obj:`save`, :obj:`load_all`, :obj:`to_parquet`, :obj:`to_hdf` .. rubric:: Notes Creates one file per locale containing that locale's chunk of each pdarray. If columns is a dictionary, the keys are used as the HDF5 dataset names. Otherwise, if no names are supplied, 0-up integers are used. By default, any existing files at path_prefix will be overwritten, unless the user specifies the 'append' mode, in which case arkouda will attempt to add as new datasets to existing files. If the wrong number of files is present or dataset names already exist, a RuntimeError is raised. .. rubric:: Examples >>> a = ak.arange(25) >>> b = ak.arange(25) >>> # Save with mapping defining dataset names >>> ak.save_all({'a': a, 'b': b}, 'path/name_prefix', file_format='Parquet') >>> # Save using names instead of mapping >>> ak.save_all([a, b], 'path/name_prefix', names=['a', 'b'], file_format='Parquet') .. py:function:: snapshot(filename) Create a snapshot of the current Arkouda namespace. All currently accessible variables containing Arkouda objects will be written to an HDF5 file. Unlike other save/load functions, this maintains the integrity of dataframes. Current Variable names are used as the dataset name when saving. :param filename: :type filename: str :param Name to use when storing file: :rtype: None .. seealso:: :obj:`ak.restore` .. py:function:: to_csv(columns: Union[Mapping[str, Union[arkouda.pdarrayclass.pdarray, arkouda.strings.Strings]], List[Union[arkouda.pdarrayclass.pdarray, arkouda.strings.Strings]]], prefix_path: str, names: Optional[List[str]] = None, col_delim: str = ',', overwrite: bool = False) Write Arkouda object(s) to CSV file(s). All CSV Files written by Arkouda include a header denoting data types of the columns. :param columns: The objects to be written to CSV file. If a mapping is used and `names` is None the keys of the mapping will be used as the dataset names. :type columns: Mapping[str, pdarray] or List[pdarray] :param prefix_path: The filename prefix to be used for saving files. Files will have _LOCALE#### appended when they are written to disk. :type prefix_path: str :param names: names of dataset to be written. Order should correspond to the order of data provided in `columns`. :type names: List[str] (Optional) :param col_delim: Defaults to ",". Value to be used to separate columns within the file. Please be sure that the value used DOES NOT appear in your dataset. :type col_delim: str :param overwrite: Defaults to False. If True, any existing files matching your provided prefix_path will be overwritten. If False, an error will be returned if existing files are found. :type overwrite: bool :rtype: None :raises ValueError: Raised if any datasets are present in all csv files or if one or more of the specified files do not exist :raises RuntimeError: Raised if one or more of the specified files cannot be opened. If `allow_errors` is true this may be raised if no values are returned from the server. :raises TypeError: Raised if we receive an unknown arkouda_type returned from the server .. seealso:: :obj:`read_csv` .. rubric:: Notes - CSV format is not currently supported by load/load_all operations - The column delimiter is expected to be the same for column names and data - Be sure that column delimiters are not found within your data. - All CSV files must delimit rows using newline (``\n``) at this time. - Unlike other file formats, CSV files store Strings as their UTF-8 format instead of storing bytes as uint(8). .. py:function:: to_hdf(columns: Union[Mapping[str, Union[arkouda.pdarrayclass.pdarray, arkouda.strings.Strings, arkouda.segarray.SegArray]], List[Union[arkouda.pdarrayclass.pdarray, arkouda.strings.Strings, arkouda.segarray.SegArray]]], prefix_path: str, names: Optional[List[str]] = None, mode: str = 'truncate', file_type: str = 'distribute') -> None Save multiple named pdarrays to HDF5 files. :param columns: Collection of arrays to save :type columns: dict or list of pdarrays :param prefix_path: Directory and filename prefix for output files :type prefix_path: str :param names: Dataset names for the pdarrays :type names: list of str :param mode: By default, truncate (overwrite) the output files if they exist. If 'append', attempt to create new dataset in existing files. :type mode: {'truncate' | 'append'} :param file_type: Default: distribute Single writes the dataset to a single file Distribute writes the dataset to a file per locale :type file_type: str ("single" | "distribute") :rtype: None :raises ValueError: Raised if (1) the lengths of columns and values differ or (2) the mode is not 'truncate' or 'append' :raises RuntimeError: Raised if a server-side error is thrown saving the pdarray .. seealso:: :obj:`to_parquet`, :obj:`load`, :obj:`load_all`, :obj:`read` .. rubric:: Notes Creates one file per locale containing that locale's chunk of each pdarray. If columns is a dictionary, the keys are used as the HDF5 dataset names. Otherwise, if no names are supplied, 0-up integers are used. By default, any existing files at path_prefix will be overwritten, unless the user specifies the 'append' mode, in which case arkouda will attempt to add as new datasets to existing files. If the wrong number of files is present or dataset names already exist, a RuntimeError is raised. .. rubric:: Examples >>> a = ak.arange(25) >>> b = ak.arange(25) >>> # Save with mapping defining dataset names >>> ak.to_hdf({'a': a, 'b': b}, 'path/name_prefix') >>> # Save using names instead of mapping >>> ak.to_hdf([a, b], 'path/name_prefix', names=['a', 'b']) .. py:function:: to_parquet(columns: Union[Mapping[str, Union[arkouda.pdarrayclass.pdarray, arkouda.strings.Strings, arkouda.segarray.SegArray]], List[Union[arkouda.pdarrayclass.pdarray, arkouda.strings.Strings, arkouda.segarray.SegArray]]], prefix_path: str, names: Optional[List[str]] = None, mode: str = 'truncate', compression: Optional[str] = None, convert_categoricals: bool = False) -> None Save multiple named pdarrays to Parquet files. :param columns: Collection of arrays to save :type columns: dict or list of pdarrays :param prefix_path: Directory and filename prefix for output files :type prefix_path: str :param names: Dataset names for the pdarrays :type names: list of str :param mode: By default, truncate (overwrite) the output files if they exist. If 'append', attempt to create new dataset in existing files. 'append' is deprecated, please use the multi-column write :type mode: {'truncate' | 'append'} :param compression: Default None Provide the compression type to use when writing the file. Supported values: snappy, gzip, brotli, zstd, lz4 convert_categoricals: bool Defaults to False Parquet requires all columns to be the same size and Categoricals don't satisfy that requirement. if set, write the equivalent Strings in place of any Categorical columns. :type compression: str (Optional) :rtype: None :raises ValueError: Raised if (1) the lengths of columns and values differ or (2) the mode is not 'truncate' or 'append' :raises RuntimeError: Raised if a server-side error is thrown saving the pdarray .. seealso:: :obj:`to_hdf`, :obj:`load`, :obj:`load_all`, :obj:`read` .. rubric:: Notes Creates one file per locale containing that locale's chunk of each pdarray. If columns is a dictionary, the keys are used as the Parquet column names. Otherwise, if no names are supplied, 0-up integers are used. By default, any existing files at path_prefix will be overwritten, unless the user specifies the 'append' mode, in which case arkouda will attempt to add as new datasets to existing files. If the wrong number of files is present or dataset names already exist, a RuntimeError is raised. .. rubric:: Examples >>> a = ak.arange(25) >>> b = ak.arange(25) >>> # Save with mapping defining dataset names >>> ak.to_parquet({'a': a, 'b': b}, 'path/name_prefix') >>> # Save using names instead of mapping >>> ak.to_parquet([a, b], 'path/name_prefix', names=['a', 'b']) .. py:function:: to_zarr(store_path: str, arr: arkouda.pdarrayclass.pdarray, chunk_shape) Writes a pdarray to disk as a Zarr store. Supports multi-dimensional pdarrays of numeric types. To use this function, ensure you have installed the blosc dependency (`make install-blosc`) and have included `ZarrMsg.chpl` in the `ServerModules.cfg` file. :param store_path: The path at which Zarr store should be written :type store_path: str :param arr: The pdarray to be written to disk :type arr: pdarray :param chunk_shape: The shape of the chunks to be used in the Zarr store :type chunk_shape: tuple :raises ValueError: Raised if the number of dimensions in the chunk shape does not match the number of dimensions in the array or if the array is not a 32 or 64 bit numeric type .. py:function:: update_hdf(columns: Union[Mapping[str, Union[arkouda.pdarrayclass.pdarray, arkouda.strings.Strings, arkouda.segarray.SegArray]], List[Union[arkouda.pdarrayclass.pdarray, arkouda.strings.Strings, arkouda.segarray.SegArray]]], prefix_path: str, names: Optional[List[str]] = None, repack: bool = True) Overwrite the datasets with name appearing in names or keys in columns if columns is a dictionary :param columns: Collection of arrays to save :type columns: dict or list of pdarrays :param prefix_path: Directory and filename prefix for output files :type prefix_path: str :param names: Dataset names for the pdarrays :type names: list of str :param repack: Default: True HDF5 does not release memory on delete. When True, the inaccessible data (that was overwritten) is removed. When False, the data remains, but is inaccessible. Setting to false will yield better performance, but will cause file sizes to expand. :type repack: bool :raises RuntimeError: Raised if a server-side error is thrown saving the datasets .. rubric:: Notes - If file does not contain File_Format attribute to indicate how it was saved, the file name is checked for _LOCALE#### to determine if it is distributed. - If the datasets provided do not exist, they will be added - Because HDF5 deletes do not release memory, this will create a copy of the file with the new data - This workflow is slightly different from `to_hdf` to prevent reading and creating a copy of the file for each dataset