DOE OSTI · 2533550
Real-time data processing for serial crystallography experiments
Abstract
We report the use of streaming data interfaces to perform fully online data processing for serial crystallography experiments, without storing intermediate data on disk. The system produces Bragg reflection intensity measurements suitable for scaling and merging, with a latency of less than 1 s per frame. Our system uses the CrystFEL software in combination with the ASAP::O data framework. In a series of user experiments at PETRA III, frames from a 16 megapixel Dectris EIGER2 X detector were searched for peaks, indexed and integrated at the maximum full-frame readout speed of 133 frames per second. The computational resources required depend on various factors, most significantly the fraction of non-blank frames ('hits'). The average single-thread processing time per frame was 242 ms for blank frames and 455 ms for hits, meaning that a single 96-core computing node was sufficient to keep up with the data, with ample headroom for unexpected throughput reductions. Further significant improvements are expected, for example by binning pixel intensities together to reduce the pixel count. We discuss the implications of real-time data processing on the `data deluge' problem from recent and future photon-science experiments, in particular on calibration requirements, computing access patterns and the need for the preservation of raw data.
Explore related subjects
Keep this discovery
Explore connections, maps & timelines
White, Thomas [Deutsches Elektronen-Synchrotron (DESY), Hamburg (Germany). Center for Data and Computing in Natural Science (CDCS)] (ORCID:0000000284070270), Schoof, Tim [Deutsches Elektronen-Synchrotron (DESY), Hamburg (Germany)], Yakubov, Sergey [Deutsches Elektronen-Synchrotron (DESY), Hamburg (Germany)] (ORCID:0000000258291280), Tolstikova, Aleksandra [Deutsches Elektronen-Synchrotron (DESY), Hamburg (Germany)] (ORCID:0000000306532123), Middendorf, Philipp [Deutsches Elektronen-Synchrotron (DESY), Hamburg (Germany). Center for Free-Electron Laser Science (CFEL)] (ORCID:0000000252438384), Karnevskiy, Mikhail [Deutsches Elektronen-Synchrotron (DESY), Hamburg (Germany)], Mariani, Valerio [SLAC National Accelerator Laboratory (SLAC), Menlo Park, CA (United States). Linac Coherent Light Source (LCLS)], Henkel, Alessandra [Deutsches Elektronen-Synchrotron (DESY), Hamburg (Germany). Center for Free-Electron Laser Science (CFEL)], Klopprogge, Bjarne [Deutsches Elektronen-Synchrotron (DESY), Hamburg (Germany). Center for Free-Electron Laser Science (CFEL); Hamburg Centre for Ultrafast Imaging (Germany)], Hannappel, Juergen [Deutsches Elektronen-Synchrotron (DESY), Hamburg (Germany)] (ORCID:0000000342974885), Oberthuer, Dominik [Deutsches Elektronen-Synchrotron (DESY), Hamburg (Germany). Center for Free-Electron Laser Science (CFEL)] (ORCID:0000000208949590), De Gennaro Aquino, Ivan [Deutsches Elektronen-Synchrotron (DESY), Hamburg (Germany). Center for Free-Electron Laser Science (CFEL)], Egorov, Dmitry [Deutsches Elektronen-Synchrotron (DESY), Hamburg (Germany). Center for Free-Electron Laser Science (CFEL)], Munke, Anna [Deutsches Elektronen-Synchrotron (DESY), Hamburg (Germany). Center for Free-Electron Laser Science (CFEL)] (ORCID:0000000255102245), Sprenger, Janina [Deutsches Elektronen-Synchrotron (DESY), Hamburg (Germany). Center for Free-Electron Laser Science (CFEL)], Pompidor, Guillaume [Deutsches Elektronen-Synchrotron (DESY), Hamburg (Germany)] (ORCID:0009000132547735), Taberman, Helena [Deutsches Elektronen-Synchrotron (DESY), Hamburg (Germany)] (ORCID:0000000175954608), Gruzinov, Andrey [Deutsches Elektronen-Synchrotron (DESY), Hamburg (Germany)], Meyer, Jan [Deutsches Elektronen-Synchrotron (DESY), Hamburg (Germany)], Hakanpää, Johanna [Deutsches Elektronen-Synchrotron (DESY), Hamburg (Germany)], Gasthuber, Martin [Deutsches Elektronen-Synchrotron (DESY), Hamburg (Germany)] (ORCID:0000000311255677). 2025-01-01. Real-time data processing for serial crystallography experiments. https://doi.org/10.1107/s2052252524011837
Cite the original work for its findings. Save a collection to share your selection of sources.