Engineering Papers⌕ Search

SEARCH · Engineering Papers

Results for “Building Dataset”

Search indexed NASA NTRS and DOE OSTI research on propulsion, heat transfer, battery materials and energy systems. Follow report and document links to the original sources.

Quote a phrase for an exact phrase match. Source license links do not imply unrestricted reuse.

At least 523 records · Page 29

An HDF5-Based Framework for the Distribution and Analysis of Ultrasonic Concrete Data

There are many commercial ultrasonic tomography devices (UTDs) available for use in nondestructive evaluation (NDE) of reinforced concrete structures. These devices emit, measure, and store ultrasonic signals typically in the 25 kHz to 5 MHz frequency range. UTDs are characterized by a composition of multiple transducers, also known as a transducer array or phased array. Often, UTDs data are in a proprietary format. Consequently, NDE research data is limited to those who have prior non-disclosure agreements or the appropriate licenses. Thus, there is a need for a proper universal data framework to exist such that proprietary file datasets for different concrete specimens can be converted, organized, and stored with relative metadata for individual or collaborative NDE research. Building upon the Hierarchical Data Format (HDF5) model, we have developed a UTD data management framework and Graphic User Interface (GUI) to promote the algorithmic reconstruction of ultrasonic data in a controlled environment for easily reproducible and publishable results.

concrete NDE↗

JHTDB-wind: a web-accessible large-eddy simulation database of a wind farm with virtual sensor querying

This paper introduces JHTDB-wind (https://turbulence.idies.jhu.edu/datasets/windfarms, last access: 11 November 2025), a publicly accessible database containing large-eddy simulation (LES) data from wind farms. Building on the framework of the Johns Hopkins Turbulence Database (JHTDB), which hosts direct numerical simulation (DNS) and some LES datasets of canonical turbulent flows, JHTDB-wind stores the 4D space–time history of the flow and provides users the ability to access and query the data via a web-based virtual sensor interface. The initial dataset comprises LES results from a large wind farm with 10×6 turbines, modeled using a filtered actuator line method, under conventionally neutral atmospheric conditions. These data comprise 1 h (hour) of flow field data (velocity, pressure, potential temperature deviation, subgrid-scale (SGS) eddy viscosity, and turbine forces, approximately 15 TB (terabytes) and wind turbine data – including both turbine-level operational quantities and blade-level aerodynamic quantities (approximately 1.3 TB) – stored in Zarr and Parquet formats, respectively. Data retrieval is facilitated by the giverny Python package, allowing remote users to query the database in Python or MATLAB (C and Fortran support are available for flow field data). This paper details the simulation setup and demonstrates data access through examples that analyze wind farm flow structures and turbine performance. The framework is extensible to future datasets, including the JHTDB-wind diurnal cycle simulation analyzed in Xiao et al. (2025).

17 WIND ENERGY↗

The NASA CloudSat/GPM Light Precipitation Validation Experiment (LPVEx)

Ground-based measurements of cool-season precipitation at mid and high latitudes (e.g., above 45 deg N/S) suggest that a significant fraction of the total precipitation volume falls in the form of light rain, i.e., at rates less than or equal to a few mm/h. These cool-season light rainfall events often originate in situations of a low-altitude (e.g., lower than 2 km) melting level and pose a significant challenge to the fidelity of all satellite-based precipitation measurements, especially those relying on the use of multifrequency passive microwave (PMW) radiometers. As a result, significant disagreements exist between satellite estimates of rainfall accumulation poleward of 45 deg. Ongoing efforts to develop, improve, and ultimately evaluate physically-based algorithms designed to detect and accurately quantify high latitude rainfall, however, suffer from a general lack of detailed, observationally-based ground validation datasets. These datasets serve as a physically consistent framework from which to test and refine algorithm assumptions, and as a means to build the library of algorithm retrieval databases in higher latitude cold-season light precipitation regimes. These databases are especially relevant to NASA's CloudSat and Global Precipitation Measurement (GPM) ground validation programs that are collecting high-latitude precipitation measurements in meteorological systems associated with frequent coolseason light precipitation events. In an effort to improve the inventory of cool-season high-latitude light precipitation databases and advance the physical process assumptions made in satellite-based precipitation retrieval algorithm development, the CloudSat and GPM mission ground validation programs collaborated with the Finnish Meteorological Institute (FMI), the University of Helsinki (UH), and Environment Canada (EC) to conduct the Light Precipitation Validation Experiment (LPVEx). The LPVEx field campaign was designed to make detailed measurements of cool-season light precipitation by leveraging existing infrastructure in the Helsinki Precipitation Testbed. LPVEx was conducted during the months of September--October, 2010 and featured coordinated ground and airborne remote sensing components designed to observe and quantify the precipitation physics associated with light rain in low-altitude melting layer environments over the Gulf of Finland and neighboring land mass surrounding Helsinki, Finland.

Petersen, Walter A.↗

Transcribing Air Traffic Control System Command Center Planning Telecons Using Cloud-Based Automatic Speech Recognition

This paper addresses the challenge of using Automatic Speech Recognition (ASR) technology to transcribe regular teleconferences that happen between FAA Air Traffic Control System Command Center (ATCSCC) planners, stakeholders and air users. These planning teleconferences (aka telecons or planning webinars) are an integral part of managing air traffic in the U.S. National Airspace System (NAS). In particular, the meetings facilitate the creation and modification of various traffic management initiatives (TMIs), that are used to regulate the flow of air traffic. This is typically a human intensive process, requiring specialists to listen to the entire meeting audio (10-20 minutes duration) and inferring the state of the NAS (e.g., weather phenomenon) that was discussed. It would be advantageous to have digital transcripts of the audio and have useful information (e.g., related to TMIs) automatically extracted from the transcripts. In this regard, we are exploring the adoption of state-of-the-art speech to text and Natural Language Processing (NLP) tools that will achieve our objective of digitizing the webinar audio. Unfortunately, the highly technical phraseology present in the audio and limited data availability for model building make ASR difficult. To overcome this challenge, we have taken the critical first step in creating a human transcription dataset from ~20 hours of speech in the ATCSCC audio with the help of subject matter experts. A novelty of our work is the creation of a ground truth transcription dataset for ATCSCC teleconference webinars, which is particularly important for Aviation domain-specific NLP tasks. Using Microsoft Speech Studio, a cloud-based ASR platform, we have fine-tuned the English pre-trained ASR models (available in speech studio) and achieved an average word error rate (WER) of 6.81%. The baseline ASR also provides a digital version of each planning webinar, making it accessible and text-searchable for future references. Additionally, the transcriptions can serve as a bridge between raw audio data and a range of text-based NLP tasks, such as named entity recognition (NER) and intent classification, potentially enhancing the digital footprint of the webinars and other connected data sources. Our work has several potential applications. Firstly, the transcriptions can be analyzed to understand the complex decision process of creating, implementing and modifying TMIs and may also contribute to TMI prediction services. Secondly, our dataset and model can be used to develop more accurate ASR systems for aviation-specific language, which can bring about digital communication in the aviation industry (and aid current “voice only” communications, which are inherently error-prone). Lastly, the transcriptions themselves can be used as a valuable resource for training other NLP models.

Stephen S. B. Clarke↗

Pilot Residential Deep Energy Retrofits and the PNNL Lab Homes

This report summarizes research investigating the technical and economic feasibility of several pilot deep energy retrofits, or retrofits that save 30% to 50% or more on a whole-house basis while increasing comfort, durability, combustion safety, and indoor air quality. The work is being conducted for the U.S. Department of Energy Building Technologies Program as part of the Building America Program. As part of the overall program, Pacific Northwest National Laboratory (PNNL) researchers are collecting and analyzing a comprehensive dataset that describes pre- and post-retrofit energy consumption, retrofit measure cost, health and comfort impacts, and other pertinent information for each home participating in the study. The research and data collection protocol includes recruitment of candidate residences, a thorough test-in audit, home energy modeling, and generation of retrofit measure recommendations, implementation of the measures, test-out, and continued evaluation. On some homes, more detailed data will be collected to disaggregate energy-consumption information. This multi-year effort began in October 2010. To date, the PNNL team has performed test-in audits on 51 homes in the marine, cold, and hot-humid climate zones, and completed 3 retrofits in Texas, 10 in Florida, and 2 in the Pacific Northwest. Two of the retrofits are anticipated to save 50% or more in energy bills and the others - savings are in the 30% to 40% range. Fourteen other retrofits are under way in the three climate zones. Metering equipment has been installed in seven of these retrofits - three in Texas, three in Florida, and one in the Pacific Northwest. This report is an interim update, providing information on the research protocol and status of the PNNL deep energy retrofit project as of December, 2011. The report also presents key findings and lessons learned, based on the body of work to date. In addition, the report summarizes the status of the PNNL Lab Homes that are new manufactured homes procured with minimal energy-efficiency specifications typical of existing homes in the region, and sited on the PNNL campus. The Lab Homes serve as a flexible test facility (the first of its kind in the Pacific Northwest) to rapidly evaluate energy-efficient and grid-smart technologies that are applicable to residential construction.

54 ENVIRONMENTAL SCIENCES↗

A cross-sectional study of the temporal evolution of electricity consumption of six commercial buildings

Current approaches to building efficiency diagnoses include conventional energy audit techniques that can be expensive and time consuming. In contrast, virtual energy audits of readily available 15-minute-interval building electricity consumption are being explored to provide quick, inexpensive, and useful insights into building operation characteristics. A cross sectional analysis of six buildings in two different climate zones provides methods for data cleaning, population-based building comparisons, and relationships (correlations) of weather and electricity consumption. Data cleaning methods have been developed to categorize and appropriately filter or correct anomalous data including outliers, missing data, and erroneous values (resulting in < 0.5% anomalies). The utility of a cross-sectional analysis of a sample set of building's electricity consumption is found through comparisons of baseload, daily consumption variance, and energy use intensity. Correlations of weather and electricity consumption 15-minute interval datasets show important relationships for the heating and cooling seasons using computed correlations of a Time-Specific-Averaged- Ordered Variable (exterior temperature) and corresponding averaged variables (electricity consumption)(TSAOV method). The TSAOV method is unique as it introduces time of day as a third variable while also minimizing randomness in both correlated variables through averaging. This study found that many of the pair-wise linear correlation analyses lacked strong relationships, prompting the development of the new TSAOV method to uncover the causal relationship between electricity and weather. We conclude that a combination of varied HVAC system operations, building thermal mass, plug load use, and building set point temperatures are likely responsible for the poor correlations in the prior studies, while the correlation of time-specific-averaged-ordered temperature and corresponding averaged variables method developed herein adequately accounts for these issues and enables discovery of strong linear pair-wise correlation R values. TSAOV correlations lay the foundation for a new approach to building studies, that mitigates plug load interferences and identifies more accurate insights into weather-energy relationship for all building types. Over all six buildings analyzed the TSAOV method reported very significant average correlations per building of 0.94 to 0.82 in magnitude. Our rigorous statistics-based methods applied to 15- minute-interval electricity data further enables virtual energy audits of buildings to quickly and inexpensively inform energy savings measures.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

Energy Efficient Stochastic-Based Deep Spiking Neural Networks for Sparse Datasets

With large deep neural networks (DNNs) neces- sary to solve complex and data-intensive problems, energy efficiency is a key bottleneck for effectively deploying DL in the real world. Deep spiking NNs have gained much research at- tention recently due to the interest in building biological neural networks and the availability of neuromorphic platforms, which can be orders of magnitude more energy efficient compared to CPUs and GPUs. Although spiking NNs have proven to be an efficient technique for solving many machine learning and computer vision problems, to the best of our knowledge, this is the first attempt to adapt spiking NNs to sparse datasets. In this paper, we study the behaviour of spiking NNs in handling NLP datasets and the sparsity in their data representation. Then, we propose a novel framework for spiking NN using the concept of stochastic computing. Specifically, instead of generating spike trains with firing rates proportional to the intensity of each value in the feature set separately, the whole feature set is treated as a distribution function and a stochastic spiking train that follow this distribution is generated. This framework reduces the connectivity between NN layers from O(N) to O(logN). Also, it encodes input data differently and make suitable to handle sparse datasets. Finally, the framework achieves high energy efficiency since it uses Integrate and Fire neurons same as conventional spiking NNs. The results show that our proposed stochastic-based SNN achieves nearly the same accuracy as the original DNN on MNIST dataset, and it has better performance than state- of-the-art SNN. Besides that stochastic-based SNN is energy efficient, where the fully connected DNN, the conventional SNN, and the data normalized SNN consume 38.24, 1.83, and 1.85 -times more energy than the stochastic-based SNN, respectively. For sparse datasets, including IMDb and In-House clinical datasets, stochastic-based SNN achieves performance comparable to that of the conventional DNN. However, the conventional spiking NN has a significant decline in classification accuracy.

Alawad, Mohammed↗

Solving Seismic Wave Equations on Variable Velocity Models With Fourier Neural Operator

Here, in the study of subsurface seismic imaging, solving the acoustic wave equation is a pivotal component in existing models. The advancement of deep learning (DL) enables solving partial differential equations (PDEs), including wave equations, by applying neural networks to identify the mapping between the inputs and the solution. This approach can be faster than traditional numerical methods when numerous instances are to be solved. Previous works that concentrate on solving the wave equation by neural networks consider either a single velocity model or multiple simple velocity models, which is restricted in practice. Instead, inspired by the idea of operator learning, this work leverages the Fourier neural operator (FNO) to effectively learn the frequency domain seismic wavefields under the context of variable velocity models. We also propose a new framework paralleled FNO (PFNO) for efficiently training the FNO-based solver given multiple source locations and frequencies. Numerical experiments demonstrate the high accuracy of both FNO and PFNO with complicated velocity models in the OpenFWI datasets. Furthermore, the cross-dataset generalization test verifies that PFNO adapts to out-of-distribution velocity models. Finally, PFNO admits higher computational efficiency on large-scale testing datasets than the traditional finite-difference method. The aforementioned advantages endow the FNO-based solver with the potential to build powerful models for research on seismic waves.

58 GEOSCIENCES↗

DOE Joint Genome Institute Strategic Planning for the Genomic Sciences. Report from the May 30–31, 2012, Workshop

The U.S. Department of Energy (DOE) Joint Genome Institute ( JGI) Strategic Planning for the Genomic Sciences workshop was convened by the DOE Office of Science’s Biological and Environmental Research Program (BER) on May 30–31, 2012. The goal was to explore DOE JGI’s role in addressing DOE mission critical scientific questions and in contributing data and knowledge to enable a new generation of systems biology research. DOE JGI has played a leadership role in genome sequencing, providing a foundation for complex biological studies. As DOE JGI moves forward into the next decade(s), it will have a continued and expanded role in genome sequencing. A major opportunity lies before DOE JGI as it seeks to build on its sequencing strength by providing high-throughput, “value-added” science that can be integrated with massive sequence datasets to accelerate the science underpinning DOE missions in bioenergy and the environment.

59 BASIC BIOLOGICAL SCIENCES↗

Fault Dataset Development for VAV Terminal Units: Damper and Airflow Sensor Faults

This report analyzes field data to understand the impact of faults in variable air volume (VAV) terminal units on building indoor conditions and heating, ventilation, and air-conditioning (HVAC) system operations. The Oak Ridge National Laboratory (ORNL) team conducted field tests at the commercial building test facility known as ORNL’s Flexible Research Platform (FRP) building. Two specific faults-damper malfunctions and airflow sensor inaccuracies- were implemented, as these are common faults in VAV terminal units and can significantly impact HVAC system performance by increasing energy consumption, causing occupant discomfort, and raising operational costs.

32 ENERGY CONSERVATION, CONSUMPTION, AND UTILIZATI↗

An AI-Enabled Chat Bot for DuraMAT

The DuraMAT Data Hub has evolved to meet the ever expanding research demands by integrating a chatbot interface that enhances FAIR compliance and simplifies discovery across our growing archive of projects and datasets. With the Data Hubs increasing use by both consortium researchers and the international community, traditional navigation methods have become unwieldy. Building on last year's feasibility study, we now report the successful implementation of the chatbot system, detailing its architecture and demonstrating its ability to deliver a more intuitive and rewarding user experience.

14 SOLAR ENERGY↗

The SkyView Data Model: Handling Heterogeneous Data Using Dynamic Software

The SkyView Virtual Telescope allows users to query over 75 distinct datasets of astronomical imagery including more than 190 distinct surveys. These holdings are extraordinarily diverse ranging over more than 15 orders of magnitude in frequency and5 orders of magnitude in resolution. To accommodate supporting these data, SkyView has developed a simple data model for surveys. Nor surprisingly, given the diverse origins of the datasets included, some surveys do not easily fit within this model. The conventional approach to dealing with such outliers is to build a layer on top of the actual data which more faithfully adheres to the data model. With SkyView we have used a somewhat alternative approach. We allow the data model to define new behaviors in the SkyView code itself either by customizing the use of existing capabilities or pointing to new classes that have non-standard behavior. This approach is feasible since in modern computer languages the code that is run can be selected dynamically with new classes completely overriding traditional behavior. This paper describes this approach and considers how it may become more useful as our software structures become increasingly dynamically linked.

Thomas McGlynn↗

SatCORPS Global Cloud Composite (GCC): the Design and Delivery of A High Quality, High Resolution, Global Cloud Product Available in Near-Real Time

The NASA Satellite ClOud and Radiation Property retrieval System (SatCORPS) supports the development of an analysis ready and cloud-optimized data transformation pipeline and geospatial service enablement of a global cloud composite (GCC) product derived from global geostationary satellite imagery. This geospatial service will be available at high temporal and spatial resolution via the SatCORPS web mapping application for visualization and analysis as well as direct ingestion to common geospatial software and custom programming. The resulting global cloud composite products from the processing pipeline can then be geospatially-service enabled as ArcGIS Image Services and Open Geospatial Consortium (OGC) Web Mapping/Coverage Services for visualization and analysis via a web mapping application and common geospatial software. Near real time global observations are created through the composition of five geostationary satellites that provides modelling and forecasting communities with the capability to provide high quality and timely information to start the projection process. The Global Cloud Composite product combines information from geostationary satellites, GOES-16, GOES-17, Himawari-8, Meteosat-11 and Meteosat-9 to create a single global composite netcdf file and images using the different products within the netcdf file. The SatCORPS team, though our Global Cloud Composite (GCC) product and web-based visualization tools including Geographical Information System (GIS) services provide near real time global cloud product information to both automated processes and traditional web users that is timely and high quality derived from geostationary satellites. The Global Cloud Composite product takes advantage of the scalable processing resources provided by the AWS batch service to provide new composites every thirty minutes. Because information from each of the low earth orbiting satellites is available on schedules tuned to the specific satellite, the processing algorithm temporally composites the final dataset as each satellite’s information becomes available. The SatCORPS team has leveraged our experience using Amazon Web Services (AWS) to build a low latency high availability tool that allows end users both human and automated to acquire high quality and high-resolution Geostationary Earth Orbiting (GEO) information at zero cost to the end user. This presentation will describe how we architected and implemented the service as well as lessons learned based on our experiences both developing and operating the system. The lessons learned include how we integrated multiple services including Amazon Batch, Amazon S3 and Amazon Lambda service to create a low cost but high-performance processing system that is capable of identifying and processing the most appropriate satellite overpass information into global cloud composites. We will also describe our web-based tools including our Geographic Information System that can be used for visualization and analysis. The products from the processing can be geospatially-service enabled as ArcGIS Image Services and Open Geospatial Consortium (OGC) Web Mapping/Coverage Services for visualization and analysis via a web mapping application and common geospatial software. The SatCORPS Global Composite Cloud product provides sophisticated global composited cloud research products with very low latency that we see that as filling a rapidly growing need in the research and modelling community with no up-front nor ongoing costs associated with downloading or using the information.

AWS AMCE SMCE GCC SATCORPS GLOBAL CLOUD COMPOSITE ↗

Potential impacts of climate change on the built environment: ASHRAE climate zones, building codes and national energy efficiency

Statement of the Problem: ASHRAE releases updates to 90.1 “Energy Standard for Buildings except Low-Rise Residential Buildings” every three years resulting in a 3.7%-17.3% increase in energy efficiency for buildings with each release. This is adopted by or informs building codes in nations across the globe, is the National Standard for the US, and individual states elect which release year of the standard they will enforce. These codes are built upon Standard 169 “Climatic Data for Building Design Standards,” the latest 2017 release of which defines climate zones based on 8, 118 weather stations throughout the world and data from the past 8-25 years. This data may not be indicative of the weather that new buildings built today, will see during their upcoming 30-120 year lifespan. Methodology & Theoretical Orientation: Using more modern, high-resolution datasets from climate satellites, IPCC climate models (PCM and HadGCM), high performance computing resources (Titan) and new capabilities for clustering and optimization the authors briefly analyzed different methods for redefining climate zones. Using bottom-up analysis of multiple meteorological variables which were the subject matter, experts selected as being important to energy consumption, rather than the heating/cooling degree days currently used. Findings: We analyzed the accuracy of redefined climate zones, compared to current climate zones and how the climate zones moved under different climate change scenarios, and quantified the accuracy of these methods on a local level, at a national scale for the US. Conclusion & Significance: There is likely to be a significant annual, national energy and cost (billions USD) savings that could be realized by adjusting climate zones to take into account anticipated trends or scenarios in regional weather patterns.

New, Joshua Ryan↗

CORAL: A framework for rigorous self-validated data modeling and integrative, reproducible data analysis

Abstract Background Many organizations face challenges in managing and analyzing data, especially when relevant datasets arise from multiple sources and methods. Analyzing heterogeneous datasets and additional derived data requires rigorous tracking of their interrelationships and provenance. This task has long been a Grand Challenge of data science and has more recently been formalized in the FAIR principles: that all data objects be Findable, Accessible, Interoperable, and Reusable, both for machines and for people. Adherence to these principles is necessary for proper stewardship of information, for testing regulatory compliance, for measuring the efficiency of processes, and for facilitating reuse of data-analytical frameworks. Findings We present the Contextual Ontology-based Repository Analysis Library (CORAL), a platform that greatly facilitates adherence to all 4 of the FAIR principles, including the especially difficult challenge of making heterogeneous datasets Interoperable and Reusable across all parts of a large, long-lasting organization. To achieve this, CORAL's data model requires that data generators extensively document the context for all data, and our tools maintain that context throughout the entire analysis pipeline. CORAL also features a web interface for data generators to upload and explore data, as well as a Jupyter notebook interface for data analysts, both backed by a common API. Conclusions CORAL enables organizations to build FAIR data types on the fly as they are needed, avoiding the expense of bespoke data modeling. CORAL provides a uniquely powerful platform to enable integrative cross-dataset analyses, generating deeper insights than are possible using traditional analysis tools.

97 MATHEMATICS AND COMPUTING↗

M & V Shootout: Setting the Stage For Testing the Performance of New Energy Baseline

Trustworthy savings calculations are critical to convincing investors in energy efficiency projects of the benefit and cost-effectiveness of such investments and their ability to replace or defer supply-side capital investments. However, today’s methods for measurement and verification (M&V) of energy savings constitute a significant portion of the total costs of efficiency projects. They also require time-consuming data acquisition and often do not deliver results until years after the program period has ended. A spectrum of savings calculation approaches are used, with some relying more heavily on measured data and others relying more heavily on estimated or modeled data, or stipulated information. The rising availability of “smart” meters, combined with new analytical approaches to quantifying savings, has opened the door to conducting M&V more quickly and at lower cost, with comparable or improved accuracy. Energy management and information systems (EMIS) technologies, not only enable significant site energy savings, but are also beginning to offer M&V capabilities. This paper expands recent analyses of public-domain, whole-building M&V methods, focusing on more novel baseline modeling approaches that leverage interval meter data. We detail a testing procedure and metrics to assess the performance of these new approaches using a large test dataset. We also provide conclusions regarding the accuracy, cost, and time trade-offs between more traditional M&V and these emerging streamlined methods. Finally, we discuss the potential evolution of M&V to better support the energy efficiency industry through low-cost approaches, and the long-term agenda for validation of building energy analytics.

Touzani, Samir↗

Piloting the Use of Energy Policy Data to Drive Market Action

Information is only useful when put to use. Across the U.S., an ever-growing number of state and local jurisdictions are implementing building performance reporting laws regarding building energy and water use in the commercial and multifamily sectors. This wealth of performance information is not yet being deployed to its full potential to drive smarter business decisions and increase energy savings. Recognizing that this information can help advance energy efficiency in cities, in 2015 the Institute for Market Transformation (IMT) partnered with New York City and the District of Columbia and local groups, NYCEEC and the District of Columbia Sustainable Energy Utility, on a three-year project to understand and document how, exactly, this data can and should be integrated to reap an array of benefits for cities, energy efficiency service providers, utilities, and building owners. This paper examines how local governments, utilities, and program implementers can replicate the success of DC and NYC. It gathers the findings from the Cities’ and their partners’ efforts to fully understand the datasets, overcome integration challenges, and apply the data to visual engagement, strategic outreach, evidence-based policymaking, and retrofit project identification and deployment. The paper goes beyond theoretical discussion by outlining practical application of data using specific examples from NYC and DC, summarizing challenges and findings, and highlighting next steps for other jurisdictions to learn from and build off the work done in those jurisdictions.

Cities↗

Uncovering interpretable relationships in high-dimensional scientific data through function preserving projections

Abstract In many fields of science and engineering, we frequently encounter experiments or simulations datasets that describe the behavior of complex systems and uncovering human interpretable patterns between their inputs and outputs via exploratory data analysis is essential for building intuition and facilitating discovery. Often, we resort to 2D embeddings for examining these high-dimensional relationships (e.g. dimensionality reduction). However, most existing embedding methods treat the dimensions as coordinates for samples in a high-dimensional space, which fail to capture the potential functional relationships, and the few methods that do take function into consideration either only focus on linear patterns or produce non-linear embeddings that are hard to interpret. To address these challenges, we proposed function preserving projections (FPP), which construct 2D linear embeddings optimized to reveal interpretable yet potentially non-linear patterns between the domain and the range of a high-dimensional function. The intuition here is that humans are good at understanding potentially non-linear patterns in 2D but unable to interpret non-linear mapping from high-dimensional space to 2D. Therefore, we should restrict the projection to linear but not the pattern we are seeking. Using FPP on real-world datasets, one can obtain fundamentally new insights about high-dimensional relationships in extremely large datasets that could not be processed with existing dimension reduction methods.

97 MATHEMATICS AND COMPUTING↗