Python Automation and Machine Learning for EM and ICs

In modern data management workflows, the ability to efficiently monitor file changes across complex directory structures is essential for ensuring traceability, version control, and audit readiness.

In method one, this code here presents a lightweight, Python-based script designed to recursively traverse directories, capture file paths, and log detailed metadata including last modified timestamps and cumulative modification counts. The script ensures that only new or updated files are recorded, reducing redundancy and enhancing performance. Furthermore, the output is dynamically sorted by the most recent modification time, allowing users to quickly identify recent activity. This tool serves as a practical solution for developers, analysts, and system administrators seeking a transparent and automated method for tracking file changes and maintaining historical logs in structured environments. Seecode.

In method two, to streamline directory monitoring and file logging tasks, a Python script was developed to automate the inspection and documentation of folder contents. The script begins by verifying whether a specified directory exists and is non-empty. Upon confirmation, it appends the current date (in YYYY-MM-DD format) to a log file, enabling timestamped tracking of folder activity. It then enumerates all items within the target directory, distinguishing between files and subfolders. For each file detected, the script checks whether its full path is already recorded in the log file; if not, it appends the new path to prevent duplication. Folder names are printed for user reference but are not logged. This approach ensures an up-to-date and non-redundant record of file paths in active directories, facilitating version control, system audits, or data management processes in a reproducible and transparent manner.code.

On the other hand, Python scripts (codeto add all lines into the text file orcodeto only add new lines into the text file) can automate a two-stage file discovery process across designated directories based on time-based modification criteria. In the first step, the script, for instance, scans two specified root directories to identify their direct subfolders that have been either modified or created within the last x days. These qualifying subfolders are then recursively searched in the second step to find all files that have been modified or generated within the last y days. The script compiles a list of the full paths of these files and writes them into a text file. This method enables efficient and targeted identification of recently updated data across layered directory structures, which is particularly useful for time-sensitive monitoring, automated data collection, or version tracking in research and production environments.

Python scripts can also automate the recursive listing of files and subfolders within a SharePoint directory, leveraging the SharePoint REST API. For instance, a script (code) serves as a basic implementation that navigates through a specified SharePoint folder, printing out the names and sizes of all contained files and subfolders in a hierarchical manner. Another script (code) builds upon this foundation by adding two key enhancements: robust exception handling and persistent logging. It captures and logs errors that may occur when accessing certain folders or files, thus allowing the script to continue running even if individual requests fail. Additionally, the second script records newly discovered file paths and metadata (file size) into an external text file (file_list.txt), while ensuring that duplicate entries are avoided by comparing each new line with previously logged content. Together, these scripts provide a flexible and reliable solution for tracking and documenting file structures within SharePoint repositories, which is particularly useful for data management and auditing tasks in engineering and research environments.