Accessing files with open()
Write mode, 'w'
file = open('shopping.txt', 'w')
file.write("apples")
file.write("bananas")
file.write("cherries")
file.close()
Read mode, 'r'
file = open('shopping.txt', 'r')
print(file.read())
file.close()
applesbananascherries
Context manager
A context manager is an object that defines the context of a with
statement.
The built-in function open()
is the most commonly used context manager in python.
It manages both opening and closing a file so the context of the code block includes an open file handle.
with open('shopping.txt', 'w') as file:
file.write("apples")
file.write("bananas")
file.write("cherries")
This code will always automatically close your files and should always be used when accessing files.
with open('shopping.txt', 'r') as file:
print(file.read())
applesbananascherries
What is this file object?
The open()
function returns different object types, depending on the mode.
for mode in ['r', 'w', 'rb', 'wb']:
with open('shopping.txt', mode) as f:
print(f"{mode}: {f}")
r: <_io.TextIOWrapper name='shopping.txt' mode='r' encoding='UTF-8'>
w: <_io.TextIOWrapper name='shopping.txt' mode='w' encoding='UTF-8'>
rb: <_io.BufferedReader name='shopping.txt'>
wb: <_io.BufferedWriter name='shopping.txt'>
Notice that text file objects are the default and that they default to utf-8
encoding,
In most cases the default read (r
) and write (w
) modes are all you will ever need.
However, if you are working with files with binary formats, then it is possible.
Markup languages such as HTML are all about human-readability
Creating line-oriented files
Notice that the result we got when we wrote data into our 'shopping.txt'
file was all in one line.
applesbananascherries
We can add literal newline characters into our files to write line-oriented data.
shopping = ['apples', 'bananas', 'cherries']
with open('shopping.txt', 'w') as file:
for item in shopping:
file.write(f"{item}\n")
This produces a result which is clearly easier to parse.
apples
bananas
cherries
Using print()
with files
The print()
function takes an optional file
argument.
shopping = ['apples', 'bananas', 'cherries']
with open('shopping.txt', 'w') as file:
for item in shopping:
print(item, file=file)
This produce an equivalent result.
apples
bananas
cherries
This is because print()
takes a end
argument which defaults to a newline character, \n
.
Parsing line-oriented data
With a line-oriented file, we can extract lines, one at a time using the readline()
method.
Each call will read data from the file up to and including the next '\n'
character.
If we print the output, then we end up adding an extra newline character.
with open('shopping.txt', 'r') as file:
apples = file.readline()
print(apples) # 'apples\n'
Calling readlines()
will generate a list of all the line-oriented data within a file, with the '\n'
characters intact.
with open('shopping.txt', 'r') as file:
data = file.readlines()
print(data[1]) # 'bananas\n'
Files are iterable
Files are iterable, so we can convert them to a list by passing a file object into the list
constructor.
with open('shopping.txt', 'r') as file:
shopping = list(file)
print(shopping)
['apples\n', 'bananas\n', 'cherries\n']
Looping is also possible. Which will keep the memory footprint lower.
with open('shopping.txt', 'r') as file:
for line in file:
print(line)
Again, in both cases the newline characters are kept.
Parsing data in other ways
To remove the newline characters we can call str.split('\n')
on the entire file contents.
with open('shopping.txt', 'r') as file:
shopping = file.read().split('\n'):
print(shopping)
['apples', 'bananas', 'cherries']
However, if our file is empty this will give a single item list containing one empty string.
This is why the string method splitlines()
exists.
with open('shopping.txt', 'r') as file:
shopping = file.read().splitlines()
print(shopping)
['apples', 'bananas', 'cherries']
Using pathlib
Many complex issues arise when managing files and folders for a cross-platform application.
An excellent tool in the python standard library is the pathlib
module.
The core pathlib
tool is the Path
class which provides an object-oriented interface representing locations in the filesystem.
from pathlib import Path
my_path = Path('folder1', 'folder2', 'filename.txt')
print(repr(my_path))
print(repr(my_path.absolute()))
PosixPath('folder1/folder2/filename.txt')
PosixPath('/home/graeme/Teaching/GAMR1520/folder1/folder2/filename.txt')
Notice we are using repr()
above to print the string representation of the path object.
Note that the result will be platform dependent and the file may or may not exist at this point.
mkdir()
and touch()
We can see that the file doesn’t exist.
my_path = Path('folder1', 'folder2', 'filename.txt')
print(my_path.exists())
False
We can create the containing folders and the file.
my_path.mkdir(parents=True)
my_path.touch()
print(my_path.exists)
True
iter_dir()
, is_dir()
and is_file()
We can iterate over the file system.
from pathlib import Path
here = Path('.')
for p in here.iterdir():
if p.is_dir():
print(str(p).ljust(20), "(folder)", sep="\t")
if p.is_file():
print(str(p).ljust(20), "(file)", sep="\t")
shopping.txt (file)
list.txt (file)
string.txt (file)
folder1 (folder)
my_experiments (folder)
script.py (file)
Joining paths
The Path.joinpath()
method can be used to create new Path
objects.
root = Path('folder')
my_path = root.joinpath('filename.txt')
Alternatively, we can use the slash operator (/
) to join paths.
This is equivalent to the above.
root = Path('folder')
my_path = root / 'filename.txt'
As we shall see, this makes manipulating files and generating directory structures very simple.
Path.open()
Conveniently, the Path
object includes an open
method which calls the built-in open
function for us.
So we can create and populate multiple files with a simplified loop.
root = Path('folder')
root.mkdir()
all_lists = {
'fruit': ['apples', 'bananas', 'cherries'],
'colours': ['aquamarine', 'blue', 'cyan'],
'animals': ['armadillo', 'baboon', 'cat']
}
for title, my_list in all_lists.items():
my_path = root / f'{title}.txt'
with my_path.open('w') as my_file:
for item in my_list:
print(item, file=my_file)
What if I have more complex data?
animals.py
animals = [{'name': 'Anteater', 'description': 'Eats ants'},
{'name': 'Bear', 'description': 'Grizzly'},
{'name': 'Chimp', 'description': 'Chump'},
{'name': 'Dog', 'description': 'Friend'}]
animals.csv
name,description
Anteater,Eats ants
Bear,Grizzly
Chimp,Chump
Dog,Friend
animals.json
[{"name": "Anteater", "description": "Eats ants"}, {"name": "Bear", "description": "Grizzly"}, {"name": "Chimp", "description": "Chump"}, {"name": "Dog", "description": "Friend"}]
Write as CSV
The csv
module takes some getting used to, but is really very simple.
from pathlib import Path
from csv import DictWriter
animals = [{'name': 'Anteater', 'description': 'Eats ants'},
{'name': 'Bear', 'description': 'Grizzly'},
{'name': 'Chimp', 'description': 'Chump'},
{'name': 'Dog', 'description': 'Friend'}]
path = Path('animals.csv')
with path.open('w') as file:
writer = DictWriter(file, fieldnames=['name', 'description'])
writer.writeheader()
writer.writerows(animals)
Write as JSON
import json
from pathlib import Path
import json
animals = [{'name': 'Anteater', 'description': 'Eats ants'},
{'name': 'Bear', 'description': 'Grizzly'},
{'name': 'Chimp', 'description': 'Chump'},
{'name': 'Dog', 'description': 'Friend'}]
path = Path('animals.json')
with path.open('w') as file:
json.dump(animals, file, indent=2)
JSON to CSV
Say you need to read from a JSON file and write into a CSV file.
This requires json.load()
and a csv.DictWriter
object.
One with
clause can open both files.
from pathlib import Path
from csv import DictWriter
import json
inpath = Path('animals.json')
outpath = Path('animals_from_json.csv')
with inpath.open('r') as infile, outpath.open('w') as outfile:
animals = json.load(infile)
writer = DictWriter(outfile, fieldnames=['name', 'description'])
writer.writeheader()
writer.writerows(animals)
We need to write the CSV header row before we write all the data.
CSV to JSON
Converting in the other direction requires a csv.DictReader
object and json.dump()
.
Notice the
csv.DictReader
object will use the header row to determine the field names. Also, because it’s iterable, we can just pass it intolist()
to load all the data.
from pathlib import Path
from csv import DictReader
import json
inpath = Path('animals.csv')
outpath = Path('animals_from_csv.json')
with inpath.open('r') as infile, outpath.open('w') as outfile:
animals = list(DictReader(infile))
json.dump(animals, outfile, indent=2)