Support to Convert .fit Results to CSV (or any format)

[kevScheuer]

This request is to merge a script and set of classes that will allow any Amptools-based analysis to convert their .fit results into a comma-separated value (CSV) file. Several plotters already exist for analyzing fit results per bin, and these are very well suited for analyzing the angular distributions, but mass-independent fits must "stitch" together their fit results to observe any behavior of the amplitudes and phases across mass bins. In addition, the 100s of fit results produced by bootstrap or randomized fits have no standard way to be aggregated. This CSV converter is designed to fill this gap in the analysis process. Below I've provided a short description for each component added.

convert_to_csv

This is the primary script that users will interact with. A user with several fit results result_1.fit, result_2.fit... can simply execute

user@ifarm:~$ convert_to_csv -i dir/result_*.fit

and a CSV will be made where each row corresponds to the .fit file, and the columns indicate AmpTools fit outputs, parameters, intensities, and phase differences.

This CSV can then be read into a Python Pandas dataframe, ROOT tree or dataframe, or used by practically any programming language, and then plotted. The script is designed to be as generic as possible, so that any AmpTools-based analysis can use it. Listed are some more highlighted features of the script

• Info about the data (number of events, t_bin info, mass bin info, etc.) the fit was run on is extracted with the --data-file flag. It will read the associated data (with optional weights and/or background) files of the result and extract the info to a CSV file
  • Different reactions are supported via the --lower-vertex-indices flag. This tells the ROOTDataConverter which 4-vector indices correspond to the upper or lower vertex, thus allowing the correct calculation of the mass and $-t$ info
    • By default, the reaction is assumed to simply be a recoil proton
• Can produce covariance, correlation, and normalization integral matrices
• Identifies coherent sums according to the amplitude naming scheme, which can be explicitly set via --naming-scheme
  • See AmplitudeParser for more details

FitConverter

Handles the .fit -> .csv conversion. This class stores:

• Standard fit outputs (likelihood, events, status codes)
• Parameters
• Production Coefficients
• Intensities of unique amplitudes (see here for more explanation)
• Coherent sums of amplitudes by quantum number (see AmplitudeParser below)
• Phase Differences between amplitudes

Currently supports .fit -> .csv conversion, but can easily be expanded to any file format desired. This is because all the results of interest are stored in various maps, and so writing to CSV is as easy as iterating over the maps.

ROOTDataConverter

This class is responsible for extracting the PWA-related information from a ROOT file. It stores:

• Bin edges, centers, averages, and RMS values for $-t$, beam energy, and upper vertex masses
• Number of events and detector efficiency

Just like the FitConverter, any file format beyond CSV can be used. To get the info, the class uses the data and monte carlo files associated with the fit. If available, it also properly incorporates event weights or background files. As discussed above, to calculate the mass and $-t$ info, the user specifies the 4-vector indices.

AmplitudeParser

This was the biggest hurdle for generalizing the converter. A lot of times we are not just interested in the individual amplitudes and phases, but their (in)coherent sums, like "total reflectivity contribution" or "behavior of JL waves summed over the spin-projections". The problem is that these sums are typically defined manually, because the amplitudes (and thus their quantum numbers) are user defined. The only way to identify them for grouping is by identifying the naming scheme of the amplitude, but not everyone uses the same scheme.

This class tries to identify the amplitude naming scheme used, and defines a set of possible sums based off the quantum numbers given in the scheme. It currently supports:

• JLme - the current recommended generic format
• eJPmL - used for some vector-pseudosalar analyses
• Lme - common scheme for 2-pseudoscalar analysis
  but can be easily extended to other schemes by users.

Updates from previous version

For those using the older standalone version of this script shown in the last tutorial, I figure its worth it to list some key differences:

• Fast - pure C++, instead of the clunky python -> subprocess -> ROOT interpreter being done before
• Easy Start - no longer have to setup python envs and ROOT paths, it's all immediately available in halld_sim now
• Generalized - data files don't have to be separately called, and all types of analyses should be supported now.

[gluex]

Test status for this pull request: SUCCESS

Summary: /work/halld/pull_request_test/halld_sim^csv_converter/tests/summary.txt
Logs: /work/halld/pull_request_test/halld_sim^csv_converter/tests/log

Build log: /work/halld/pull_request_test/halld_sim^csv_converter/make_csv_converter.log
Build report: /work/halld/pull_request_test/halld_sim^csv_converter/report_csv_converter.txt
Location of build: /work/halld/pull_request_test/halld_sim^csv_converter

[gluex]

Test status for this pull request: SUCCESS

Summary: summary.txt
Logs: results/log

Build log: make_csv_converter.log
Build report: report_csv_converter.txt

[lihaoahil]

Thanks for developing this converter — I tested it on a few fit outputs and it works for standard partial-wave-style fits where the amplitude naming follows the usual expected structure.

I did run into a crash for an SDME fit with a slightly different but still valid AmpTools configuration, where the intensity function has multiple factors in one sum. This is different from the mass-independent PWA-style fits that I tested successfully, though similar structures may also appear in some mass-dependent PWA fits. In my case the .fit file contains amplitudes like

amplitude twoPS0::SDMEext::kstar KStarHyperonExtended ...
amplitude twoPS0::SDMEext::kstar BreitWigner ...

The crash happens in FitConverter::sumAmpNamesAreConstrained(), where unique_sum_amp is assumed to contain "::" before being split into sum and amp. With my fit, the debug output shows

DEBUG sumAmpNamesAreConstrained: unique_sum_amp = SDMEext, sum = SDMEext, amp = DMEext, shared_terms.size() = 0

So unique_sum_amp.find("::") returns npos, and then substr(find("::") + 2) produces a malformed amplitude name. The following termList("", sum, amp) returns an empty vector, and the code later accesses shared_terms[0], which causes the segfault.

I think this is mostly a robustness issue rather than a problem with the general converter logic. It would be helpful if the converter could fall back to full amplitude names when the shortened naming scheme cannot be inferred reliably, rather than crashing.

I have made the test .fit files for both PWA and doubleSDME and related input available here: /w/halld-scshelf2101/home/haoli/analysis/testIO_kpi/test_Kevin