Python File I/O

Python File I/O#

Learn about Python file operations. More specifically, opening a file, reading from it, writing into it, closing it, and various file methods that you should be aware of.

Files#

Files are named locations on disk to store related information. They are used to permanently store data in a non-volatile memory (e.g. hard disk).

Since Random Access Memory (RAM) is volatile (which loses its data when the computer is turned off), we use files for future use of the data by permanently storing them.

When we want to read from or write to a file, we need to open it first. When we are done, it needs to be closed so that the resources that are tied with the file are freed.

Hence, in Python, a file operation takes place in the following order:

Open a file

Close the file

Write into files (perform operation)

Read contents of files (perform operation)

Opening Files in Python#

Python has a built-in open() function to open a file. This function returns a file object, also called a handle, as it is used to read or modify the file accordingly.

>>> f = open("test.txt")  # open file in current directory
>>> f = open("C:/Python99/README.txt")   # specifying full path

We can specify the mode while opening a file. In mode, we specify whether we want to read r, write w or append a to the file. We can also specify if we want to open the file in text mode or binary mode.

The default is reading in text mode. In this mode, we get strings when reading from the file.

On the other hand, binary mode returns bytes and this is the mode to be used when dealing with non-text files like images or executable files.

Mode	Description
`r`	Read -Opens a file for reading only. The file pointer is placed at the beginning of the file. This is the default mode.
`t`	Text - Opens in text mode. (default).
`b`	Binary - Opens in binary mode (e.g. images).
`x`	Create - Opens a file for exclusive creation. If the file already exists, the operation fails.
`rb`	Opens a file for reading only in binary format. The file pointer is placed at the beginning of the file. This is the default mode.
`r+`	Opens a file for both reading and writing. The file pointer placed at the beginning of the file.
`rb+`	Opens a file for both reading and writing in binary format. The file pointer placed at the beginning of the file.
`w`	Write - Opens a file for writing only. Overwrites the file if the file exists. If the file does not exist, creates a new file for writing.
`wb`	Opens a file for writing only in binary format. Overwrites the file if the file exists. If the file does not exist, creates a new file for writing.
`w+`	Opens a file for both writing and reading. Overwrites the existing file if the file exists. If the file does not exist, creates a new file for reading and writing.
`wb+`	Opens a file for both writing and reading in binary format. Overwrites the existing file if the file exists. If the file does not exist, creates a new file for reading and writing.
`a`	Append - Opens a file for appending. The file pointer is at the end of the file if the file exists. That is, the file is in the append mode. If the file does not exist, it creates a new file for writing.
`ab`	Opens a file for appending in binary format. The file pointer is at the end of the file if the file exists. That is, the file is in the append mode. If the file does not exist, it creates a new file for writing.
`a+`	Opens a file for both appending and reading. The file pointer is at the end of the file if the file exists. The file opens in the append mode. If the file does not exist, it creates a new file for reading and writing.
`ab+`	Opens a file for both appending and reading in binary format. The file pointer is at the end of the file if the file exists. The file opens in the append mode. If the file does not exist, it creates a new file for reading and writing.

f = open("test.txt")   # equivalent to 'r' or 'rt'
print(f)               # <_io.TextIOWrapper name='test.txt' mode='r' encoding='cp1252'>

<_io.TextIOWrapper name='test.txt' mode='r' encoding='cp1252'>

Opened file has different reading methods: read(), readline, readlines. An opened file has to be closed with close() method.

f = open("test.txt",'w')  # write in text mode
print(f)

<_io.TextIOWrapper name='test.txt' mode='w' encoding='cp1252'>

f = open("logo.png",'r+b')  # read and write in binary mode

Unlike other languages, the character a does not imply the number 97 until it is encoded using ASCII (or other equivalent encodings).

Moreover, the default encoding is platform dependent. In windows, it is cp1252 but utf-8 in Linux.

So, we must not also rely on the default encoding or else our code will behave differently in different platforms.

Hence, when working with files in text mode, it is highly recommended to specify the encoding type.

f = open("test.txt", mode='r', encoding='utf-8')

Closing Files in Python#

When we are done with performing operations on the file, we need to properly close the file.

Closing a file will free up the resources that were tied with the file. It is done using the close() method available in Python.

Python has a garbage collector to clean up unreferenced objects but we must not rely on it to close the file.

f = open("test.txt", encoding = 'utf-8')
# perform file operations
f.close()

This method is not entirely safe. If an exception occurs when we are performing some operation with the file, the code exits without closing the file.

A safer way is to use a try-finally block.

try:
    f = open("test.txt", encoding = 'utf-8')
    # perform file operations
finally:
    f.close()

This way, we are guaranteeing that the file is properly closed even if an exception is raised that causes program flow to stop.

The best way to close a file is by using the with statement. This ensures that the file is closed when the block inside the with statement is exited.

We don’t need to explicitly call the close() method. It is done internally.

>>>with open("test.txt", encoding = 'utf-8') as f:
   # perform file operations

The file Object Attributes#

file.closed - Returns true if file is closed, false otherwise.
file.mode - Returns access mode with which file was opened.
file.name - Returns name of the file.

# Open a file
data = open("data.txt", "wb")
print ("Name of the file: ", data.name)
print ("Closed or not : ", data.closed)
print ("Opening mode : ", data.mode)
data.close()  #closed data.txt file

Name of the file:  data.txt
Closed or not :  False
Opening mode :  wb

Writing to Files in Python#

In order to write into a file in Python, we need to open it in write w, append a or exclusive creation x mode.

We need to be careful with the w mode, as it will overwrite into the file if it already exists. Due to this, all the previous data are erased.

Writing a string or sequence of bytes (for binary files) is done using the write() method. This method returns the number of characters written to the file.

with open("test_1.txt",'w',encoding = 'utf-8') as f:
    f.write("my first file\n")
    f.write("This file\n\n")
    f.write("contains three lines\n")

This program will create a new file named test_1.txt in the current directory if it does not exist. If it does exist, it is overwritten.

We must include the newline characters ourselves to distinguish the different lines.

with open("test_2.txt",'w',encoding = 'utf-8') as f:
    f.write("This is file\n")
    f.write("my\n")
    f.write("first file\n")

Let us append a some text to the file we have been reading:

with open("test_2.txt",'a',encoding = 'utf-8') as f:
    f.write('This text has to be appended at the end')

# open a file in current directory
data = open("data_1.txt", "w") # "w" write in text mode,
data.write("Welcome to Python Tutorial")
print("done")
data.close()

done

Reading Files in Python#

To read a file in Python, we must open the file in reading r mode.

There are various methods available for this purpose. We can use the read(size) method to read in the size number of data. If the size parameter is not specified, it reads and returns up to the end of the file.

We can read the text_1.txt file we wrote in the above section in the following way:

f = open("test.txt",'r',encoding = 'utf-8')
txt = f.read()  # read all the characters in the file
print(type(txt))
print(txt)
f.close()

<class 'str'>

f = open("test_1.txt",'r',encoding = 'utf-8')
f.read(8)  # read the first 8 data characters

'my first'

f.read(5)  # read the next 5 data characters

' file'

f.read()  # read in the rest till end of file

'\nThis file\n\ncontains three lines\n'

f.read()  # further reading returns empty sting

''

We can see that the read() method returns a newline as '\n'. Once the end of the file is reached, we get an empty string on further reading.

We can change our current file cursor (position) using the seek() method. Similarly, the tell() method returns our current position (in number of bytes).

f.tell()    # get the current file position

f.seek(0)   # bring file cursor to initial position

print(f.read())  # read the entire file

my first file
This file

contains three lines

We can read a file line-by-line using a for loop. This is both efficient and fast.

f = open("test_1.txt",'r',encoding = 'utf-8')
for line in f:
    print(line, end = '')

my first file
This file

contains three lines

In this program, the lines in the file itself include a newline character \n. So, we use the end parameter of the print() function to avoid two newlines when printing.

Alternatively, we can use the readline() method to read individual lines of a file. This method reads a file till the newline, including the newline character.

f.seek(0)  # bring file cursor to initial position
f.readline()

'my first file\n'

f.readline()

'This file\n'

f.readline()

'\n'

f.readline()

'contains three lines\n'

Lastly, the readlines() method returns a list of remaining lines of the entire file. All these reading methods return empty values when the end of file (EOF) is reached.

f.seek(0)  # bring file cursor to initial position
f.readlines()

['my first file\n', 'This file\n', '\n', 'contains three lines\n']

Another way to get all the lines as a list is using splitlines()

f.seek(0)  # bring file cursor to initial position
f.read().splitlines()

['my first file', 'This file', '', 'contains three lines']

# Open a file
data = open("data_1.txt", "r+")
file_data = data.read(27) # read 3.375 byte only
full_data = data.read()   # read all byte into file from last cursor
print(file_data)
print(full_data)
data.close()

Welcome to Dr. Milan Parmar
's Python Tutorial

File Positions#

The tell() method tells you the current position within the file; in other words, the next read or write will occur at that many bytes from the beginning of the file.

The seek(offset[, from]) method changes the current file position. The offset argument indicates the number of bytes to be moved. The from argument specifies the reference position from where the bytes are to be moved.

If from is set to 0, the beginning of the file is used as the reference position. If it is set to 1, the current position is used as the reference position. If it is set to 2 then the end of the file would be taken as the reference position.

# Open a file
data = open("data_1.txt", "r+")
file_data = data.read(27) # read 18 byte only
print("current position after reading 27 byte :",data.tell())
data.seek(0) #here current position set to 0 (starting of file)
full_data = data.read() #read all byte
print(file_data)
print(full_data)
print("position after reading file : ",data.tell())
data.close()

current position after reading 27 byte : 26
Welcome to Python Tutorial
Welcome to Python Tutorial
position after reading file :  26

Python File Methods#

There are various methods available with the file object. Some of them have been used in the above examples.

Here is the complete list of methods in text mode with a brief description:

Method	Description
`close()`	Closes an opened file. It has no effect if the file is already closed.
`detach()`	Separates the underlying binary buffer from the `TextIOBase` and returns it.
`fileno()`	Returns an integer number (file descriptor) of the file.
`flush()`	Flushes the write buffer of the file stream.
`isatty()`	Returns `True` if the file stream is interactive.
`read(n)`	Reads at most `n` characters from the file. Reads till end of file if it is negative or `None`.
`readable()`	Returns `True` if the file stream can be read from.
`readline(n=-1)`	Reads and returns one line from the file. Reads in at most `n` bytes if specified.
`readlines(n=-1)`	Reads and returns a list of lines from the file. Reads in at most `n` bytes/characters if specified.
`seek(offset,from=SEEK_SET)`	Changes the file position to `offset` bytes, in reference to `from` (start, current, end).
`seekable()`	Returns `True` if the file stream supports random access.
`tell()`	Returns the current file location.
`truncate(size=None)`	Resizes the file stream to `size` bytes. If `size` is not specified, resizes to current location…
`writable()`	Returns `True` if the file stream can be written to.
`write(s)`	Writes the string `s` to the file and returns the number of characters written…
`writelines(lines)`	Writes a list of `lines` to the file…

Deleting Files#

We have seen in previous section, how to make and remove a directory using os module (04_Python_Functions ➞ 007_Python_Function_Module ➞ Python Built-In Modules). Again now, if we want to remove a file we use os module.

import os
os.remove('example.txt')

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-29-c3f637926d7e> in <module>
      1 import os
----> 2 os.remove('example.txt')

FileNotFoundError: [WinError 2] The system cannot find the file specified: 'example.txt'

If the file does not exist, the remove method will raise an error, so it is good to use a condition like this:

import os
if os.path.exists('./files/example.txt'):
    os.remove('./files/example.txt')
else:
    print('The file does not exist')

The file does not exist

File Types#

File with txt Extension#

File with txt extension is a very common form of data and we have covered it in the previous section. Let us move to the JSON file.

File with json Extension#

JSON stands for JavaScript Object Notation. Actually, it is a stringified JavaScript object or Python dictionary.

# dictionary
person_dct= {
    "name":"Milaan",
    "country":"England",
    "city":"London",
    "skills":["Python", "MATLAB","R"]
}
# JSON: A string form a dictionary
person_json = "{'name': 'Milaan', 'country': 'England', 'city': 'London', 'skills': ['Python', 'MATLAB','R']}"

# we use three quotes and make it multiple line to make it more readable
person_json = '''{
    "name":"Milaan",
    "country":"England",
    "city":"London",
    "skills":["Python", "MATLAB","R"]
}'''

Changing JSON to Dictionary#

To change a JSON to a dictionary, first we import the json module and then we use loads method.

import json
# JSON
person_json = '''{
    "name":"Milaan",
    "country":"England",
    "city":"London",
    "skills":["Python", "MATLAB","R"]
}'''
# let's change JSON to dictionary
person_dct = json.loads(person_json)
print(type(person_dct))
print(person_dct)
print(person_dct['name'])

<class 'dict'>
{'name': 'Milaan', 'country': 'England', 'city': 'London', 'skills': ['Python', 'MATLAB', 'R']}
Milaan

Changing Dictionary to JSON#

To change a dictionary to a JSON we use dumps method from the json module.

import json
# python dictionary
person = {
    "name":"Milaan",
    "country":"England",
    "city":"London",
    "skills":["Python", "MATLAB","R"]
}
# let's convert it to  json
person_json = json.dumps(person, indent=4) # indent could be 2, 4, 8. It beautifies the json
print(type(person_json))
print(person_json)

# when you print it, it does not have the quote, but actually it is a string
# JSON does not have type, it is a string type.

<class 'str'>
{
    "name": "Milaan",
    "country": "England",
    "city": "London",
    "skills": [
        "Python",
        "MATLAB",
        "R"
    ]
}

Saving as JSON File#

We can also save our data as a json file. Let us save it as a json file using the following steps. For writing a json file, we use the json.dump() method, it can take dictionary, output file, ensure_ascii and indent.

import json
# python dictionary
person = {
    "name":"Milaan",
    "country":"England",
    "city":"London",
    "skills":["Python", "MATLAB","R"]
}
with open('json_example.json', 'w', encoding='utf-8') as f:
    json.dump(person, f, ensure_ascii=False, indent=4)

In the code above, we use encoding and indentation. Indentation makes the json file easy to read.

File with csv Extension#

CSV stands for Comma Separated Values. CSV is a simple file format used to store tabular data, such as a spreadsheet or database. CSV is a very common data format in data science.

For example, create csv_example.csv in your working directory with the following contents:

"name","country","city","skills"
"Milaan","England","London","Python"

import csv
with open('csv_example.csv') as f:
    csv_reader = csv.reader(f, delimiter=',') # w use, reader method to read csv
    line_count = 0
    for row in csv_reader:
        if line_count == 0:
            print(f'Column names are :{", ".join(row)}')
            line_count += 1
        else:
            print(
                f'\t{row[0]} is a teachers. He lives in {row[1]}, {row[2]}.')
            line_count += 1
    print(f'Number of lines:  {line_count}')

Column names are :name, country, city, skills
	Milaan is a teachers. He lives in England, London.
Number of lines:  2

File with xlsx Extension#

To read excel files we need to install xlrd package. We will cover this after we cover package installing using pip.

import xlrd
excel_book = xlrd.open_workbook('sample.xls)
print(excel_book.nsheets)
print(excel_book.sheet_names)

File with xml Extension#

XML is another structured data format which looks like HTML. In XML the tags are not predefined. The first line is an XML declaration. The person tag is the root of the XML. The person has a gender attribute.

<?xml version="1.0"?>
<person gender="female">
  <name>Asabeneh</name>
  <country>Finland</country>
  <city>Helsinki</city>
  <skills>
    <skill>JavaScrip</skill>
    <skill>React</skill>
    <skill>Python</skill>
  </skills>
</person>

For more information on how to read an XML file check the documentation

import xml.etree.ElementTree as ET
tree = ET.parse('xml_example.xml')
root = tree.getroot()
print('Root tag:', root.tag)
print('Attribute:', root.attrib)
for child in root:
    print('field: ', child.tag)

Root tag: person
Attribute: {'gender': 'male'}
field:  name
field:  country
field:  city
field:  skills