Tag: size

The code below takes a directory of xml files and analyzes them in a csv file.
Currently, scanning around 60 XML is fast and the output is a Csv file of around 250 MB.

This is a real big file and the reason is that the columns are repeated. I repeat the columns for the reason that each element must have the information. in reading is on cases where Z048 had multiple lines of setdata, this is why the other columns in red had to be repeated.

I plan to increase the number of xml to 5 KB, which means the csv file will be relatively large.

Ask this question to maybe get an answer if the size of my csv file can be reduced in one way or another. Even though when coding, I tried to think that I wanted my code to be fast and not produce large csv files.

from xml.etree import ElementTree as ET
from collections import defaultdict
import csv
from pathlib import Path

directory = 'path to a folder with xml files'

with open('output.csv', 'w', newline='') as f:
    writer = csv.writer(f)

    headers = ('id', 'service_code', 'rational', 'qualify', 'description_num', 'description_txt', 'set_data_xin', 'set_data_xax', 'set_data_value', 'set_data_x')
    writer.writerow(headers)

    xml_files_list = list(map(str, Path(directory).glob('**/*.xml')))
    print(xml_files_list)
    for xml_file in xml_files_list:
        tree = ET.parse(xml_file)
        root = tree.getroot()

        start_nodes = root.findall('.//START')
        for sn in start_nodes:
            row = defaultdict(str)

            repeated_values = dict()
            for k,v in sn.attrib.items():
                repeated_values(k) = v

            for rn in sn.findall('.//Rational'):
                repeated_values('rational') = rn.text

            for qu in sn.findall('.//Qualify'):
                repeated_values('qualify') = qu.text

            for ds in sn.findall('.//Description'):
                repeated_values('description_txt') = ds.text
                repeated_values('description_num') = ds.attrib('num')

            for st in sn.findall('.//SetData'):
                for k,v in st.attrib.items():
                    row('set_data_'+ str(k)) = v
                for key in repeated_values.keys():
                    row(key) = repeated_values(key)
                row_data = (row(i) for i in headers)
                writer.writerow(row_data)
                row = defaultdict(str)