Zohner, PorterJ | Engineering Papers

DOE OSTI2024

Deeplynx Dag Repository

The DeepLynx DAG repository will contain several Airflow DAGs (Directed Acyclic Graphs) which will be used in the context of DeepLynx's deployed Apache Airflow instance. These DAGs will be used for multiple data management tasks for DeepLynx data, including but not limited to: - bringing data from various sources and tools into DeepLynx - managing sequential data workflows, such as running Python scripts on data to perform analysis and returning the results to DeepLynx - performing any necessary transformation or pre-processing on data coming into DeepLynx from external sources or out of DeepLynx to go to external applications

Brownlee, JarenM.↗

DOE OSTI2024

Deeplynx Airflow Provider Package

The DeepLynx Airflow Provider Package is a python package used to interact with the data warehouse DeepLynx when using the workflow orchestration tool Apache Airflow. This python package is packaged together using the airflow package standard so that it can be easily installed and used in any Apache Airflow environment. This package is meant to encapsulate the DeepLynx API for use in Airflow so that any interactions with DeepLynx that a user may want to use in their Airflow workflow can be easily accomplished using this provider package. This allows us to develop, implement, and test our DeepLynx-Airflow interactions in one provider package repository, and then easily install and use this package in any airflow instance. This DeepLynx Airflow Provider Package will be used extensively by the DeepLynx DAG repository.

Cavaluzzi, JackM↗