Reading and Writing Files with NumPy in Python

Josh Included in Python

Published 2025-02-26 Updated 2025-03-02 2141 words 11 minutes views

The cover image is the official NumPy logo, retrieved from GitHub on February 26, 2025.

Introduction

When it comes to numerical and matrix operations in Python, we must mention NumPy, a module widely favored by most programmers. NumPy supports high-dimensional, large-scale multi-dimensional array and matrix operations, and it provides an extensive library of mathematical functions for array operations, making numerical computations in Python faster and easier. However, one drawback is that NumPy can only run on a CPU and cannot leverage GPU acceleration.

This article focuses solely on file reading and writing in NumPy. For other functionalities, please refer to additional articles or the official documentation https://numpy.org/doc/stable/reference/.

Supported File Formats

NumPy reads and writes files in binary mode, supporting both binary and plain text formats. Common formats include NumPy’s dedicated binary formats .npy and .npz, as well as .bin for binary storage and .txt or .csv for plain text storage.

`.npy` Format

The .npy format is designed for storing and loading NumPy arrays (ndarray), including their data, shape, and data type (dtype). Since .npy files store a single NumPy array, they are optimized for space efficiency and faster reading speeds compared to .txt and .csv files due to their binary structure.

`.npz` Format

The .npz format functions similarly to .npy, but it allows storing multiple NumPy arrays in a single file. Additionally, the .savez_compressed() function can be used to compress the stored data.

`.bin` Format

Although .bin is also a binary format, it differs from .npy in that it only stores raw data without metadata such as shape and data type. Therefore, when reading .bin files, the shape must always be manually specified.

Writing to Files

NumPy provides various methods for saving numerical data and matrices to files. This section introduces the following methods: .save(), .savez(), .savez_compressed(), .savetxt(), and .tofile().

First, we import the NumPy module and assume we have two NumPy arrays, arr1 and arr2.

1
2
3
4
import numpy as np

arr1 = np.array([10, 20, 30])
arr2 = np.array([[1, 2], [3, 4]])

`.save()`

Usage:

Saves a single NumPy array in the .npy binary format.

Syntax:

.save(file, arr, allow_pickle=True)

Parameters:

file: The filename, string, or file path where the array will be saved.
arr: The NumPy array to be saved.
allow_pickle: Boolean (optional). If True, allows using Python’s pickle to save object arrays. However, not all pickle data is compatible across different Python versions. Default is True.

Example:

1
np.save("array.npy", arr1) # Saves arr1 to "array.npy"

`.savez()`

Usage:

Saves multiple arrays without compression into a single .npz file.

Syntax:

.savez(file, *args, allow_pickle=True, **kwds)

Parameters:

file: The filename, string, or file path where the arrays will be saved.
args: The arrays to be saved (optional). If args is used, the arrays will be named "arr_0", "arr_1", etc.
allow_pickle: Boolean (optional). If True, allows using Python’s pickle to save object arrays. However, not all pickle data is compatible across different Python versions. Default is True.
kwds: Named arrays (optional). Each array is stored with its corresponding custom name.

Example:

1
np.savez("arrays.npz", first=arr1, second=arr2) # Saves multiple arrays with custom names

`.savez_compressed()`

Usage:

Save multiple arrays in a compressed format to a single .npz file.

Syntax:

.savez_compressed(file, *args, allow_pickle=True, **kwds)

Parameters:

file: The filename, string, or file path where the arrays will be saved.
args: The arrays to be saved (optional). If args is used, the arrays will be named "arr_0", "arr_1", etc.
allow_pickle: Boolean (optional). If True, allows using Python’s pickle to save object arrays. However, not all pickle data is compatible across different Python versions. Default is True.
kwds: Named arrays (optional). Each array is stored with its corresponding custom name.

Example:

1
np.savez_compressed("arrays_compressed.npz", first=arr1, second=arr2)

`.savetxt()`

Usage:

Saves an array as a plain text file, typically in .txt or .csv format.

Syntax:

.savetxt(fname, X, fmt='%.18e', delimiter=' ', newline='\n', header='', footer='', comments='# ', encoding=None)

Parameters:

fname: The filename, string, or file path where the arrays will be saved. If the file extension is .gz, it will automatically be saved in gzip-compressed format, and .loadtxt() will also automatically decompress and read the gzip file.
X: A one-dimensional or two-dimensional array to be saved as a plain text file.
fmt: A string or sequence of strings (optional). This parameter is complex, please refer to the official documentation https://numpy.org/doc/stable/reference/generated/numpy.savetxt.html.
delimiter: The delimiter used to separate values in each row. It can be a string or character (optional).
newline: The newline character used to separate lines. It can be a string or character (optional).
header: A string written at the beginning of the file.
footer: A string written at the end of the file.
comments: The comment character(s) used before the header and footer. The default is "# ".
encoding: None or a string (optional). Specifies the encoding used for the output file. The default is "latin1".

Example:

1
np.savetxt("array.txt", arr1, delimiter=",")

`.tofile()`

Usage:

Saves data in binary or plain text format. The default is binary format.

Syntax:

.tofile(fid, sep='', format='%s')

Parameters:

fid: The filename, string, or file path where the data will be saved.
sep: A string specifying the separator between array items for text output.
format: A string defining the output format for text files.

Example:

1
2
arr1.tofile("array.bin") # Saves in binary format by default
arr2.tofile("array.txt", sep=",", format="%d") # Saves in plain text format

Reading Files

The following introduces how to use .load(), .loadtxt(), .genfromtxt(), .fromregex(), .fromstring(), .fromfile(), and other methods to read files into NumPy.

First, we load the NumPy module.

1
import numpy as np

`.load()`

Usage:

Reads arrays or pickle objects from .npy, .npz, or pickle files.

Syntax:

.load(file, mmap_mode=None, allow_pickle=False, fix_imports=True, encoding='ASCII', *, max_header_size=10000)

Parameters:

file: The filename, string, or file path to be read.
mmap_mode: None, r+, r, w+, or c (optional). This parameter is complex, refer to the official documentation https://numpy.org/doc/stable/reference/generated/numpy.load.html.
allow_pickle: Boolean (optional). Whether to allow loading pickle objects stored in .npy files. Default is False.
fix_imports: Boolean (optional). Only useful when loading a pickle file created in Python 2 on Python 3.
encoding: "latin1", "ASCII", or "bytes" (optional). The encoding used when reading Python 2 strings. Default is "ASCII".
max_header_size: Integer (optional). Maximum allowed header size. Large headers may be unsafe to load.

Example:

1
2
3
4
5
loaded_arr = np.load("array.npy") # Load `.npy` file
print(loaded_arr)

loaded_npz = np.load("arrays.npz") # Load `.npz` file
print(loaded_npz["first"]) # Access the "first" array inside `arrays.npz`

`.loadtxt()`

Usage:

Reads text files.

Syntax:

.loadtxt(fname, dtype=<class 'float'>, comments='#', delimiter=None, converters=None, skiprows=0, usecols=None, unpack=False, ndmin=0, encoding=None, max_rows=None, *, quotechar=None, like=None)

Parameters:

fname: The filename, string, file path, or generator. If the file extension is .gz, it will be automatically decompressed when reading the gzip-compressed file.
dtype: Data type (optional). Default is float.
comments: Comment character (string or None, optional). Default is "# ".
delimiter: Delimiter character (string, optional). Default is a space.
converters: Dictionary (optional). Default is None.
skiprows: Number of initial rows to skip, including comments. Default is 0.
usecols: Specifies which columns to read (integer, optional). 0 represents the first column, and so on. Default is None, meaning all columns are read.
unpack: Boolean (optional). If True, the returned array is transposed. Default is False.
ndmin: Integer (optional). The array will have at least ndmin dimensions. Valid values are 0 (default), 1, or 2.
encoding: String (optional). Default is None. This parameter is complex, refer to the official documentation https://numpy.org/doc/stable/reference/generated/numpy.loadtxt.html.
max_rows: Integer (optional). Reads up to max_rows rows after skiprows, excluding empty lines.
quotechar: Unicode character or None (optional). Character used to mark the beginning and end of quoted elements. Default is None.
like: array_like object (optional). Refer to the official documentation https://numpy.org/doc/stable/reference/generated/numpy.loadtxt.html.

Example:

1
2
loaded_txt = np.loadtxt("array.txt", delimiter=",", dtype=int)
print(loaded_txt)

`.genfromtxt()`

Usage:

Load data from a plain text file and handle missing values as specified.

Syntax:

.genfromtxt(fname, dtype=<class 'float'>, comments='#', delimiter=None, skip_header=0, skip_footer=0, converters=None, missing_values=None, filling_values=None, usecols=None, names=None, excludelist=None, deletechars=" !#$%&'()*+, -./:;<=>?@[\\]^{|}~", replace_space='_', autostrip=False, case_sensitive=True, defaultfmt='f%i', unpack=None, usemask=False, loose=True, invalid_raise=True, max_rows=None, encoding=None, *, ndmin=0, like=None)

Parameters:

fname: The filename, string, file path, or generator. If the file extension is .gz or .bz2, it will be automatically decompressed when reading the compressed file.
dtype: The data type (optional). Default is float.
comment: The comment character, a string (optional).
delimiter: The delimiter character, which can be a string, integer, or sequence (optional).
skip_header: Integer (optional). The number of lines to skip at the beginning of the file.
skip_footer: Integer (optional). The number of lines to skip at the end of the file.
converters: Variable (optional). A set of functions that convert column data into values; it can also provide default values for missing data.
missing_values: Variable (optional). The string(s) corresponding to missing values.
filling_values: Variable (optional). The default value for missing data.
usecols: The columns to read, an integer (optional). 0 represents the first column, and so on. Default is None, meaning all columns are read.
names: None, True, a string, or a sequence (optional). If True, column names are read from the first line after skip_header.
excludelist: Sequence (optional). A list of names to exclude from names.
deletechars: String (optional). Defines the list of invalid characters to remove from names.
replace_space: Character (optional). The character used to replace spaces in variable names. Default is "_".
autostrip: Boolean (optional). Whether to automatically strip spaces from variables.
case_sensitive: True, False, "upper", or "lower" (optional). If True, column names are case-sensitive. If False or "upper", column names are converted to uppercase. If "lower", column names are converted to lowercase.
defaultfmt: String (optional). Defines the format for default column names.
unpack: Boolean (optional). If True, the returned array is transposed. Default is False.
usemask: Boolean (optional). If True, returns a masked array. If False, returns a regular array.
loose: Boolean (optional). If True, invalid values will not trigger errors.
invalid_raise: Boolean (optional). If True, an exception is raised when inconsistent row counts are detected. If False, a warning is issued, and problematic rows are skipped.
max_rows: Integer (optional). The maximum number of rows to read. Cannot be used with skip_footer and must be at least 1.
encoding: String (optional). The encoding used to decode the input file. Default is None.
ndmin: Integer (optional). The minimum number of dimensions in the returned array. Valid values are 0 (default), 1, or 2.
like: array_like object (optional). Refer to the official documentation https://numpy.org/doc/stable/reference/generated/numpy.genfromtxt.html.

Example:

1
2
loaded_gen = np.genfromtxt("array.txt", delimiter=",", dtype=int, filling_values=-1)
print(loaded_gen)

`.fromregex()`

Usage:

Read an array from a text file that matches a regular expression.

Syntax:

.fromregex(file, regexp, dtype, encoding=None)

Parameters:

file: The filename, string, or file path to be read.
regexp: String or regular expression (regexp) used to parse the file.
dtype: Data type (dtype) or a list of data types.
encoding: String (optional). Specifies the encoding used to decode the input file.

Example:

1
2
3
4
5
import io
data = io.StringIO("123 abc\n456 def\n789 ghi")
pattern = r"(\d+)\s(\w+)"  # Parsing "Number + Space + Text"
loaded_regex = np.fromregex(data, pattern, dtype=[("num", int), ("text", "U10")])
print(loaded_regex)

`.fromstring()`

Usage:

Initialize a new array from text data in a string.

Syntax:

.fromstring(string, dtype=float, count=-1, *, sep, like=None)

Parameters:

string: The string containing the data.
dtype: Data type (optional). Default is float.
count: Integer (optional). The number of elements to read from the string. If count is negative, the entire string will be read.
sep: The separator, a string (optional).
like: array_like object (optional). Refer to the official documentation https://numpy.org/doc/stable/reference/generated/numpy.fromstring.html.

Example:

1
2
3
s = "1,2,3,4,5"
loaded_str = np.fromstring(s, dtype=int, sep=",")
print(loaded_str)

`.fromfile()`

Usage:

Create an array from data in a text file or binary format file.

Syntax:

.fromfile(file, dtype=float, count=-1, sep='', offset=0, *, like=None)

Parameters:

file: The filename, string, or file path to be read.
dtype: Data type (optional).
count: The number of elements to read, -1 means to read all.
sep: The delimiter. If it’s a text file, the default is "".
offset: Integer, the offset relative to the current position in the file, default is 0.
like: array_like object (optional). Refer to the official documentation https://numpy.org/doc/stable/reference/generated/numpy.fromfile.html.

Example:

1
2
3
4
5
loaded_bin = np.fromfile("array.bin", dtype=int) # Read binary data
print(loaded_bin)

loaded_txt = np.fromfile("array.txt", dtype=int, sep=",") # Read text file data
print(loaded_txt)

Conclusion

Getting a thorough understanding of NumPy’s input and output is quite a task, and there are certainly many commands and parameters that I don’t fully comprehend or have never used. However, through learning the commands listed above, I believe I now have a bit more knowledge of NumPy.

References

plusone. (January 6, 2018). [Day18]Numpy檔案輸入與輸出！IT邦幫忙. Retrieved on February 27, 2025, from https://ithelp.ithome.com.tw/articles/10196167
為自己coding. (April 22, 2021). 給自己的Python小筆記 - Numpy如何讀寫檔案? - NumPy獨有的npy和npz二進制檔案格式和text(.txt)檔案格式 - 讀取/寫入教學. Matters. Retrieved on February 27, 2025, from https://matters.town/a/k50fv7lzpioq
Ashe child 由羔羊. (June 28, 2021). Python： NumPy 的二進制檔案 .npy. 羔羊的實驗紀錄簿. Retrieved on February 27, 2025, from https://yang10001.yia.app/2021/06/28/python：-numpy-的二進制檔案-npy/
NumPy. (November 30, 2024). Wikipedia, The Free Encyclopedia. Retrieved on February 27, 2025, from https://en.wikipedia.org/wiki/NumPy
Input and Output. (2024). NumPy. Retrieved on February 27, 2025, from https://numpy.org/doc/stable/reference/routines.io.html
NumPy. (February 27, 2025). Github. Retrieved on February 27, 2025, from https://github.com/numpy/numpy

Contents

Reading and Writing Files with NumPy in Python

Introduction

Supported File Formats

.npy Format

.npz Format

.bin Format

Writing to Files

.save()

.savez()

.savez_compressed()

.savetxt()

.tofile()

Reading Files

.load()

.loadtxt()

.genfromtxt()

.fromregex()

.fromstring()

.fromfile()

Conclusion

References

Related Posts

`.npy` Format

`.npz` Format

`.bin` Format

`.save()`

`.savez()`

`.savez_compressed()`

`.savetxt()`

`.tofile()`

`.load()`

`.loadtxt()`

`.genfromtxt()`

`.fromregex()`

`.fromstring()`

`.fromfile()`