arkouda.numpy.segarray

Classes

SegArray

Module Contents

class arkouda.numpy.segarray.SegArray(segments, values, lengths=None, grouping=None)

AND(x=None)

OR(x=None)

XOR(x=None)

aggregate(op, x=None)

all(x=None)

any(x=None)

append(other, axis=0)

Append other to self, either vertically (axis=0, length of resulting SegArray increases), or horizontally (axis=1, each sub-array of other appends to the corresponding sub-array of self).

Parameters:

• other (SegArray) – Array of sub-arrays to append

• axis (0 or 1) – Whether to append vertically (0) or horizontally (1). If axis=1, other must be same size as self.

Returns:

axis=0: New SegArray containing all sub-arrays axis=1: New SegArray of same length, with pairs of sub-arrays concatenated

Return type:

SegArray

append_single(x, prepend=False)

Append a single value to each sub-array.

Parameters:

x (pdarray or scalar) – Single value to append to each sub-array

Returns:

Copy of original SegArray with values from x appended to each sub-array

Return type:

SegArray

argmax(x=None)

argmin(x=None)

classmethod concat(x, axis=0, ordered=True)

Concatenate a sequence of SegArrays.

Parameters:

• x (sequence of SegArray) – The SegArrays to concatenate

• axis (0 or 1) – Select vertical (0) or horizontal (1) concatenation. If axis=1, all SegArrays must have same size.

• ordered (bool) – Must be True. This option is present for compatibility only, because unordered concatenation is not yet supported.

Returns:

The input arrays joined into one SegArray

Return type:

SegArray

copy()

Return a deep copy.

dtype

filter(filter, discard_empty: bool = False)

Filter values out of the SegArray object.

Parameters:

• filter (pdarray, list, or value) – The value/s to be filtered out of the SegArray

• discard_empty (bool) – Defaults to False. When True, empty segments are removed from the return SegArray

Return type:

SegArray

classmethod from_multi_array(m)

Construct a SegArray from a list of columns. This essentially transposes the input, resulting in an array of rows.

Parameters:

m (list of pdarray or Strings) – List of columns, the rows of which will form the sub-arrays of the output

Returns:

Array of rows of input

Return type:

SegArray

classmethod from_return_msg(rep_msg) → SegArray

get_jth(j, return_origins=True, compressed=False, default=0)

Select the j-th element of each sub-array, where possible.

Parameters:

• j (int) – The index of the value to get from each sub-array. If j is negative, it counts backwards from the end of each sub-array.

• return_origins (bool) – If True, return a logical index indicating where j is in bounds

• compressed (bool) – If False, return array is same size as self, with default value where j is out of bounds. If True, the return array only contains values where j is in bounds.

• default (scalar) – When compressed=False, the value to return when j is out of bounds for the sub-array

Returns:

valpdarray

compressed=False: The j-th value of each sub-array where j is in bounds and the default value where j is out of bounds. compressed=True: The j-th values of only the sub-arrays where j is in bounds

origin_indicespdarray, bool

A Boolean array that is True where j is in bounds for the sub-array.

Return type:

pdarray, pdarray|bool

Notes

If values are Strings, only the compressed format is supported.

get_length_n(n, return_origins=True)

Return all sub-arrays of length n, as a list of columns.

Parameters:

• n (int) – Length of sub-arrays to select

• return_origins (bool) – Return a logical index indicating which sub-arrays are length n

Returns:

columnslist of pdarray

An n-long list of pdarray, where each row is one of the n-long sub-arrays from the SegArray. The number of rows is the number of True values in the returned mask.

origin_indicespdarray, bool

Array of bool for each element of the SegArray, True where sub-array has length n.

Return type:

List of pdarray, pdarray|bool

get_ngrams(n, return_origins=True)

Return all n-grams from all sub-arrays.

Parameters:

• n (int) – Length of n-gram

• return_origins (bool) – If True, return an int64 array indicating which sub-array each returned n-gram came from.

Returns:

ngramslist of pdarray

An n-long list of pdarrays, essentially a table where each row is an n-gram.

origin_indicespdarray, int

The index of the sub-array from which the corresponding n-gram originated

Return type:

pdarray, pdarray|int

get_prefixes(n, return_origins=True, proper=True)

Return all sub-array prefixes of length n (for sub-arrays that are at least n+1 long).

Parameters:

• n (int) – Length of suffix

• return_origins (bool) – If True, return a logical index indicating which sub-arrays were long enough to return an n-prefix

• proper (bool) – If True, only return proper prefixes, i.e. from sub-arrays that are at least n+1 long. If False, allow the entire sub-array to be returned as a prefix.

Returns:

prefixeslist of pdarray

An n-long list of pdarrays, essentially a table where each row is an n-prefix. The number of rows is the number of True values in the returned mask.

origin_indicespdarray, bool

Boolean array that is True where the sub-array was long enough to return an n-suffix, False otherwise.

Return type:

List of pdarray, pdarray|bool

get_suffixes(n, return_origins=True, proper=True)

Return the n-long suffix of each sub-array, where possible.

Parameters:

• n (int) – Length of suffix

• return_origins (bool) – If True, return a logical index indicating which sub-arrays were long enough to return an n-suffix

• proper (bool) – If True, only return proper suffixes, i.e. from sub-arrays that are at least n+1 long. If False, allow the entire sub-array to be returned as a suffix.

Returns:

suffixeslist of pdarray

An n-long list of pdarrays, essentially a table where each row is an n-suffix. The number of rows is the number of True values in the returned mask.

origin_indicespdarray, bool

Boolean array that is True where the sub-array was long enough to return an n-suffix, False otherwise.

Return type:

List of pdarray, pdarray|bool

property grouping

hash() → Tuple[arkouda.numpy.pdarrayclass.pdarray, arkouda.numpy.pdarrayclass.pdarray]

Compute a 128-bit hash of each segment.

Returns:

A tuple of two int64 pdarrays. The ith hash value is the concatenation of the ith values from each array.

Return type:

Tuple[pdarray,pdarray]

intersect(other)

Computes the intersection of 2 SegArrays.

Parameters:

other (SegArray) – SegArray to compute against

Returns:

Segments are the 1d intersections of the segments of self and other

Return type:

SegArray

See also

pdarraysetops.intersect1d

Examples

>>> import arkouda as ak
>>> a = [1, 2, 3, 1, 4]
>>> b = [3, 1, 4, 5]
>>> c = [1, 3, 3, 5]
>>> d = [2, 2, 4]
>>> seg_a = ak.SegArray(ak.array([0, len(a)]), ak.array(a+b))
>>> seg_b = ak.SegArray(ak.array([0, len(c)]), ak.array(c+d))
>>> seg_a.intersect(seg_b)
SegArray([
[1 3]
[4]
])

is_registered() → bool

Check if the name of the SegArray object is registered in the Symbol Table.

Returns:

True if SegArray is registered, false if not

Return type:

bool

See also

register, unregister, attach

logger

max(x=None)

mean(x=None)

min(x=None)

property nbytes

The size of the segarray in bytes.

Returns:

The size of the segarray in bytes.

Return type:

int

property non_empty

nunique(x=None)

objType = 'SegArray'

prepend_single(x)

prod(x=None)

classmethod read_hdf(prefix_path, dataset='segarray')

Load a saved SegArray from HDF5. All arguments must match what was supplied to SegArray.save().

Parameters:

• prefix_path (str) – Directory and filename prefix

• dataset (str) – Name prefix for saved data within the HDF5 files

Return type:

SegArray

register(user_defined_name)

Register this SegArray object and underlying components with the Arkouda server.

Parameters:

user_defined_name (str) – user defined name which this SegArray object will be registered under

Returns:

The same SegArray which is now registered with the arkouda server and has an updated name. This is an in-place modification, the original is returned to support a fluid programming style. Please note you cannot register two different SegArrays with the same name.

Return type:

SegArray

Raises:

RegistrationError – Raised if the server could not register the SegArray object

Notes

Objects registered with the server are immune to deletion until they are unregistered.

See also

unregister, attach, is_registered

registered_name: str | None = None

remove_repeats(return_multiplicity=False)

Condense sequences of repeated values within a sub-array to a single value.

Parameters:

return_multiplicity (bool) – If True, also return the number of times each value was repeated.

Returns:

norepeatsSegArray

Sub-arrays with runs of repeated values replaced with single value

multiplicitySegArray

If return_multiplicity=True, this array contains the number of times each value in the returned SegArray was repeated in the original SegArray.

Return type:

Segarray, Segarray

segments

set_jth(i, j, v)

Set the j-th element of each sub-array in a subset.

Parameters:

• i (pdarray, int) – Indices of sub-arrays to set j-th element

• j (int) – Index of value to set in each sub-array. If j is negative, it counts backwards from the end of the sub-array.

• v (pdarray or scalar) – The value(s) to set. If v is a pdarray, it must have same length as i.

Raises:

ValueError – If j is out of bounds in any of the sub-arrays specified by i.

setdiff(other)

Computes the set difference of 2 SegArrays.

Parameters:

other (SegArray) – SegArray to compute against

Returns:

Segments are the 1d set difference of the segments of self and other

Return type:

SegArray

See also

pdarraysetops.setdiff1d

Examples

>>> import arkouda as ak
>>> a = [1, 2, 3, 1, 4]
>>> b = [3, 1, 4, 5]
>>> c = [1, 3, 3, 5]
>>> d = [2, 2, 4]
>>> seg_a = ak.SegArray(ak.array([0, len(a)]), ak.array(a+b))
>>> seg_b = ak.SegArray(ak.array([0, len(c)]), ak.array(c+d))
>>> seg_a.setdiff(seg_b)
SegArray([
[2 4]
[1 3 5]
])

setxor(other)

Computes the symmetric difference of 2 SegArrays.

Parameters:

other (SegArray) – SegArray to compute against

Returns:

Segments are the 1d symmetric difference of the segments of self and other

Return type:

SegArray

See also

pdarraysetops.setxor1d

Examples

>>> import arkouda as ak
>>> a = [1, 2, 3, 1, 4]
>>> b = [3, 1, 4, 5]
>>> c = [1, 3, 3, 5]
>>> d = [2, 2, 4]
>>> seg_a = ak.SegArray(ak.array([0, len(a)]), ak.array(a+b))
>>> seg_b = ak.SegArray(ak.array([0, len(c)]), ak.array(c+d))
>>> seg_a.setxor(seg_b)
SegArray([
[2 4 5]
[1 2 3 5]
])

size

sum(x=None)

to_hdf(prefix_path, dataset: str = 'segarray', mode: Literal['truncate', 'append'] = 'truncate', file_type: Literal['single', 'distribute'] = 'distribute')

Save the SegArray to HDF5. The result is a collection of HDF5 files, one file per locale of the arkouda server, where each filename starts with prefix_path.

Parameters:

• prefix_path (str) – Directory and filename prefix that all output files will share

• dataset (str) – Name prefix for saved data within the HDF5 file

• mode ({'truncate', 'append'}) – By default, truncate (overwrite) output files, if they exist. If ‘append’, add data as a new column to existing files.

• file_type ({"single", "distribute"}) – Default: “distribute” When set to single, dataset is written to a single file. When distribute, dataset is written on a file per locale. This is only supported by HDF5 files and will have no impact of Parquet Files.

Return type:

None

See also

load

to_ndarray()

Convert the array into a numpy.ndarray containing sub-arrays.

Returns:

A numpy ndarray with the same sub-arrays (also numpy.ndarray) as this array

Return type:

np.ndarray

See also

array, tolist

Examples

>>> import arkouda as ak
>>> segarr = ak.SegArray(ak.array([0, 4, 7]), ak.arange(12))
>>> segarr.to_ndarray()
array([array([0, 1, 2, 3]), array([4, 5, 6]), array([ 7,  8,  9, 10, 11])],
  dtype=object)
>>> type(segarr.to_ndarray())
<class 'numpy.ndarray'>

to_parquet(prefix_path, dataset='segarray', mode: Literal['truncate', 'append'] = 'truncate', compression: str | None = None)

Save the SegArray object to Parquet. The result is a collection of files, one file per locale of the arkouda server, where each filename starts with prefix_path. Each locale saves its chunk of the object to its corresponding file.

Parameters:

• prefix_path (str) – Directory and filename prefix that all output files share

• dataset (str) – Name of the dataset to create in files (must not already exist)

• mode ({'truncate', 'append'}) – Deprecated. Parameter kept to maintain functionality of other calls. Only Truncate supported. By default, truncate (overwrite) output files, if they exist. If ‘append’, attempt to create new dataset in existing files.

• compression (str (Optional)) – (None | “snappy” | “gzip” | “brotli” | “zstd” | “lz4”) Sets the compression type used with Parquet files

Return type:

string message indicating result of save operation

Raises:

• RuntimeError – Raised if a server-side error is thrown saving the pdarray

• ValueError – If write mode is not Truncate.

Notes

• Append mode for Parquet has been deprecated. It was not implemented for SegArray.

• The prefix_path must be visible to the arkouda server and the user must

have write permission. - Output files have names of the form <prefix_path>_LOCALE<i>, where <i> ranges from 0 to numLocales for file_type=’distribute’. - If any of the output files already exist and the mode is ‘truncate’, they will be overwritten. If the mode is ‘append’ and the number of output files is less than the number of locales or a dataset with the same name already exists, a RuntimeError will result. - Any file extension can be used.The file I/O does not rely on the extension to determine the file format.

tolist()

Convert the segarray into a list containing sub-arrays.

Returns:

A list with the same sub-arrays (also list) as this segarray

Return type:

list

See also

to_ndarray

Examples

>>> import arkouda as ak
>>> segarr = ak.SegArray(ak.array([0, 4, 7]), ak.arange(12))
>>> segarr.tolist()
[[0, 1, 2, 3], [4, 5, 6], [7, 8, 9, 10, 11]]
>>> type(segarr.tolist())
<class 'list'>

transfer(hostname: str, port: arkouda.numpy.dtypes.int_scalars)

Send a Segmented Array to a different Arkouda server.

Parameters:

• hostname (str) – The hostname where the Arkouda server intended to receive the Segmented Array is running.

• port (int_scalars) – The port to send the array over. This needs to be an open port (i.e., not one that the Arkouda server is running on). This will open up numLocales ports, each of which in succession, so will use ports of the range {port..(port+numLocales)} (e.g., running an Arkouda server of 4 nodes, port 1234 is passed as port, Arkouda will use ports 1234, 1235, 1236, and 1237 to send the array data). This port much match the port passed to the call to ak.receive_array().

Return type:

A message indicating a complete transfer

Raises:

• ValueError – Raised if the op is not within the pdarray.BinOps set

• TypeError – Raised if other is not a pdarray or the pdarray.dtype is not a supported dtype

union(other)

Computes the union of 2 SegArrays.

Parameters:

other (SegArray) – SegArray to compute against

Returns:

Segments are the 1d union of the segments of self and other

Return type:

SegArray

See also

pdarraysetops.union1d

Examples

>>> import arkouda as ak
>>> a = [1, 2, 3, 1, 4]
>>> b = [3, 1, 4, 5]
>>> c = [1, 3, 3, 5]
>>> d = [2, 2, 4]
>>> seg_a = ak.SegArray(ak.array([0, len(a)]), ak.array(a+b))
>>> seg_b = ak.SegArray(ak.array([0, len(c)]), ak.array(c+d))
>>> seg_a.union(seg_b)
SegArray([
[1 2 3 4 5]
[1 2 3 4 5]
])

unique(x=None) → SegArray

Return sub-arrays of unique values.

Parameters:

x (pdarray) – The values to unique, per group. By default, the values of this SegArray’s sub-arrays.

Returns:

Same number of sub-arrays as original SegArray, but elements in sub-array are unique and in sorted order.

Return type:

SegArray

unregister()

Unregister this SegArray object in the arkouda server which was previously registered using register() and/or attached to using attach().

Raises:

RuntimeError – Raised if the server could not unregister the SegArray object from the Symbol Table

Notes

Objects registered with the server are immune to deletion until they are unregistered.

See also

register, attach, is_registered

update_hdf(prefix_path: str, dataset: str = 'segarray', repack: bool = True)

Overwrite the dataset with the name provided with this SegArray object. If the dataset does not exist it is added.

Parameters:

• prefix_path (str) – Directory and filename prefix that all output files share

• dataset (str) – Name of the dataset to create in files

• repack (bool) – Default: True HDF5 does not release memory on delete. When True, the inaccessible data (that was overwritten) is removed. When False, the data remains, but is inaccessible. Setting to false will yield better performance, but will cause file sizes to expand.

Raises:

RuntimeError – Raised if a server-side error is thrown saving the SegArray

Notes

• If file does not contain File_Format attribute to indicate how it was saved, the file name is checked for _LOCALE#### to determine if it is distributed.

• If the dataset provided does not exist, it will be added

• Because HDF5 deletes do not release memory, this will create a copy of the file with the new data

valsize

values