26#include <nlohmann/json.hpp>
1512 void Exec(unsigned int slot)
1514 fPerThreadResults[slot]++;
1517 // Called at the end of the event loop.
1520 *fFinalResult = std::accumulate(fPerThreadResults.begin(), fPerThreadResults.end(), 0);
1523 // Called by RDataFrame to retrieve the name of this action.
1524 std::string GetActionName() const { return "MyCounter"; }
1528 ROOT::RDataFrame df(10);
1529 ROOT::RDF::RResultPtr<int> resultPtr = df.Book<>(MyCounter{df.GetNSlots()}, {});
1530 // The GetValue call triggers the event loop
1531 std::cout << "Number of processed entries: " << resultPtr.GetValue() << std::endl;
1535See the Book() method for more information and [this tutorial](https://root.cern/doc/master/df018__customActions_8C.html)
1536for a more complete example.
1538#### Injecting arbitrary code in the event loop with Foreach() and ForeachSlot()
1540Foreach() takes a callable (lambda expression, free function, functor...) and a list of columns and
1541executes the callable on the values of those columns for each event that passes all upstream selections.
1542It can be used to perform actions that are not already available in the interface. For example, the following snippet
1543evaluates the root mean square of column "x":
1545// Single-thread evaluation of RMS of column "x" using Foreach
1548df.Foreach([&sumSq, &n](double x) { ++n; sumSq += x*x; }, {"x"});
1549std::cout << "rms of x: " << std::sqrt(sumSq / n) << std::endl;
1551In multi-thread runs, users are responsible for the thread-safety of the expression passed to Foreach():
1552thread will execute the expression concurrently.
1553The code above would need to employ some resource protection mechanism to ensure non-concurrent writing of `rms`; but
1554this is probably too much head-scratch for such a simple operation.
1556ForeachSlot() can help in this situation. It is an alternative version of Foreach() for which the function takes an
1557additional "processing slot" parameter besides the columns it should be applied to. RDataFrame
1558guarantees that ForeachSlot() will invoke the user expression with different `slot` parameters for different concurrent
1559executions (see [Special helper columns: rdfentry_ and rdfslot_](\ref helper-cols) for more information on the slot parameter).
1560We can take advantage of ForeachSlot() to evaluate a thread-safe root mean square of column "x":
1562// Thread-safe evaluation of RMS of column "x" using ForeachSlot
1563ROOT::EnableImplicitMT();
1564const unsigned int nSlots = df.GetNSlots();
1565std::vector<double> sumSqs(nSlots, 0.);
1566std::vector<unsigned int> ns(nSlots, 0);
1568df.ForeachSlot([&sumSqs, &ns](unsigned int slot, double x) { sumSqs[slot] += x*x; ns[slot] += 1; }, {"x"});
1569double sumSq = std::accumulate(sumSqs.begin(), sumSqs.end(), 0.); // sum all squares
1570unsigned int n = std::accumulate(ns.begin(), ns.end(), 0); // sum all counts
1571std::cout << "rms of x: " << std::sqrt(sumSq / n) << std::endl;
1573Notice how we created one `double` variable for each processing slot and later merged their results via `std::accumulate`.
1577### Dataset joins with friend trees
1579Vertically concatenating multiple trees that have the same columns (creating a logical dataset with the same columns and
1580more rows) is trivial in RDataFrame: just pass the tree name and a list of file names to RDataFrame's constructor, or create a TChain
1581out of the desired trees and pass that to RDataFrame.
1583Horizontal concatenations of trees or chains (creating a logical dataset with the same number of rows and the union of the
1584columns of multiple trees) leverages TTree's "friend" mechanism.
1586Simple joins of trees that do not have the same number of rows are also possible with indexed friend trees (see below).
1588To use friend trees in RDataFrame, set up trees with the appropriate relationships and then instantiate an RDataFrame
1594main.AddFriend(&friend, "myFriend");
1597auto df2 = df.Filter("myFriend.MyCol == 42");
1600The same applies for TChains. Columns coming from the friend trees can be referred to by their full name, like in the example above,
1601or the friend tree name can be omitted in case the column name is not ambiguous (e.g. "MyCol" could be used instead of
1602"myFriend.MyCol" in the example above if there is no column "MyCol" in the main tree).
1604\note A common source of confusion is that trees that are written out from a multi-thread Snapshot() call will have their
1605 entries (block-wise) shuffled with respect to the original tree. Such trees cannot be used as friends of the original
1606 one: rows will be mismatched.
1608Indexed friend trees provide a way to perform simple joins of multiple trees over a common column.
1609When a certain entry in the main tree (or chain) is loaded, the friend trees (or chains) will then load an entry where the
1610"index" columns have a value identical to the one in the main one. For example, in Python:
1616# If a friend tree has an index on `commonColumn`, when the main tree loads
1617# a given row, it also loads the row of the friend tree that has the same
1618# value of `commonColumn`
1619aux_tree.BuildIndex("commonColumn")
1621mainTree.AddFriend(aux_tree)
1623df = ROOT.RDataFrame(mainTree)
1626RDataFrame supports indexed friend TTrees from ROOT v6.24 in single-thread mode and from v6.28/02 in multi-thread mode.
1628\anchor other-file-formats
1629### Reading data formats other than ROOT trees
1630RDataFrame can be interfaced with RDataSources. The ROOT::RDF::RDataSource interface defines an API that RDataFrame can use to read arbitrary columnar data formats.
1632RDataFrame calls into concrete RDataSource implementations to retrieve information about the data, retrieve (thread-local) readers or "cursors" for selected columns
1633and to advance the readers to the desired data entry.
1634Some predefined RDataSources are natively provided by ROOT such as the ROOT::RDF::RCsvDS which allows to read comma separated files:
1636auto tdf = ROOT::RDF::FromCSV("MuRun2010B.csv");
1637auto filteredEvents =
1638 tdf.Filter("Q1 * Q2 == -1")
1639 .Define("m", "sqrt(pow(E1 + E2, 2) - (pow(px1 + px2, 2) + pow(py1 + py2, 2) + pow(pz1 + pz2, 2)))");
1640auto h = filteredEvents.Histo1D("m");
1644See also FromNumpy (Python-only), FromRNTuple(), FromArrow(), FromSqlite().
1647### Computation graphs (storing and reusing sets of transformations)
1649As we saw, transformed dataframes can be stored as variables and reused multiple times to create modified versions of the dataset. This implicitly defines a **computation graph** in which
1650several paths of filtering/creation of columns are executed simultaneously, and finally aggregated results are produced.
1652RDataFrame detects when several actions use the same filter or the same defined column, and **only evaluates each
1653filter or defined column once per event**, regardless of how many times that result is used down the computation graph.
1654Objects read from each column are **built once and never copied**, for maximum efficiency.
1655When "upstream" filters are not passed, subsequent filters, temporary column expressions and actions are not evaluated,
1656so it might be advisable to put the strictest filters first in the graph.
1658\anchor representgraph
1659### Visualizing the computation graph
1660It is possible to print the computation graph from any node to obtain a [DOT (graphviz)](https://en.wikipedia.org/wiki/DOT_(graph_description_language)) representation either on the standard output
1663Invoking the function ROOT::RDF::SaveGraph() on any node that is not the head node, the computation graph of the branch
1664the node belongs to is printed. By using the head node, the entire computation graph is printed.
1666Following there is an example of usage:
1668// First, a sample computational graph is built
1669ROOT::RDataFrame df("tree", "f.root");
1671auto df2 = df.Define("x", []() { return 1; })
1672 .Filter("col0 % 1 == col0")
1673 .Filter([](int b1) { return b1 <2; }, {"cut1"})
1674 .Define("y", []() { return 1; });
1676auto count = df2.Count();
1678// Prints the graph to the rd1.dot file in the current directory
1679ROOT::RDF::SaveGraph(df, "./mydot.dot");
1680// Prints the graph to standard output
1681ROOT::RDF::SaveGraph(df);
1684The generated graph can be rendered using one of the graphviz filters, e.g. `dot`. For instance, the image below can be generated with the following command:
1686$ dot -Tpng computation_graph.dot -ocomputation_graph.png
1689\image html RDF_Graph2.png
1692### Activating RDataFrame execution logs
1694RDataFrame has experimental support for verbose logging of the event loop runtimes and other interesting related information. It is activated as follows:
1696#include <ROOT/RLogger.hxx>
1698// this increases RDF's verbosity level as long as the `verbosity` variable is in scope
1699auto verbosity = ROOT::RLogScopedVerbosity(ROOT::Detail::RDF::RDFLogChannel(), ROOT::ELogLevel::kInfo);
1706verbosity = ROOT.RLogScopedVerbosity(ROOT.Detail.RDF.RDFLogChannel(), ROOT.ELogLevel.kInfo)
1709More information (e.g. start and end of each multi-thread task) is printed using `ELogLevel.kDebug` and even more
1710(e.g. a full dump of the generated code that RDataFrame just-in-time-compiles) using `ELogLevel.kDebug+10`.
1712\anchor rdf-from-spec
1713### Creating an RDataFrame from a dataset specification file
1715RDataFrame can be created using a dataset specification JSON file:
1720df = ROOT.RDF.Experimental.FromSpec("spec.json")
1723The input dataset specification JSON file needs to be provided by the user and it describes all necessary samples and
1724their associated metadata information. The main required key is the "samples" (at least one sample is needed) and the
1725required sub-keys for each sample are "trees" and "files". Additionally, one can specify a metadata dictionary for each
1726sample in the "metadata" key.
1728A simple example for the formatting of the specification in the JSON file is the following:
1734 "trees": ["tree1", "tree2"],
1735 "files": ["file1.root", "file2.root"],
1739 "sample_category" = "data"
1743 "trees": ["tree3", "tree4"],
1744 "files": ["file3.root", "file4.root"],
1748 "sample_category" = "MC_background"
1755The metadata information from the specification file can be then accessed using the DefinePerSample function.
1756For example, to access luminosity information (stored as a double):
1759df.DefinePerSample("lumi", 'rdfsampleinfo_.GetD("lumi")')
1762or sample_category information (stored as a string):
1765df.DefinePerSample("sample_category", 'rdfsampleinfo_.GetS("sample_category")')
1768or directly the filename:
1771df.DefinePerSample("name", "rdfsampleinfo_.GetSampleName()")
1774An example implementation of the "FromSpec" method is available in tutorial: df106_HiggstoFourLeptons.py, which also
1775provides a corresponding exemplary JSON file for the dataset specification.
1778### Adding a progress bar
1780A progress bar showing the processed event statistics can be added to any RDataFrame program.
1781The event statistics include elapsed time, currently processed file, currently processed events, the rate of event processing
1782and an estimated remaining time (per file being processed). It is recorded and printed in the terminal every m events and every
1783n seconds (by default m = 1000 and n = 1). The ProgressBar can be also added when the multithread (MT) mode is enabled.
1785ProgressBar is added after creating the dataframe object (df):
1787ROOT::RDataFrame df("tree", "file.root");
1788ROOT::RDF::Experimental::AddProgressBar(df);
1791Alternatively, RDataFrame can be cast to an RNode first, giving the user more flexibility
1792For example, it can be called at any computational node, such as Filter or Define, not only the head node,
1793with no change to the ProgressBar function itself (please see the [Python interface](classROOT_1_1RDataFrame.html#python)
1794section for appropriate usage in Python):
1796ROOT::RDataFrame df("tree", "file.root");
1797auto df_1 = ROOT::RDF::RNode(df.Filter("x>1"));
1798ROOT::RDF::Experimental::AddProgressBar(df_1);
1800Examples of implemented progress bars can be seen by running [Higgs to Four Lepton tutorial](https://root.cern/doc/master/df106__HiggsToFourLeptons_8py_source.html) and [Dimuon tutorial](https://root.cern/doc/master/df102__NanoAODDimuonAnalysis_8C.html).
1802\anchor missing-values
1803### Working with missing values in the dataset
1805In certain situations a dataset might be missing one or more values at one or
1806more of its entries. For example:
1808- If the dataset is composed of multiple files and one or more files is
1809 missing one or more columns required by the analysis.
1810- When joining different datasets horizontally according to some index value
1811 (e.g. the event number), if the index does not find a match in one or more
1812 other datasets for a certain entry.
1813- If, for a certain event, a column is invalid because it results from a Snapshot
1814 with systematic variations, and that variation didn't pass its filters. For
1815 more details, see \ref snapshot-with-variations.
1817For example, suppose that column "y" does not have a value for entry 42:
1827If the RDataFrame application reads that column, for example if a Take() action
1828was requested, the default behaviour is to throw an exception indicating
1829that that column is missing an entry.
1831The following paragraphs discuss the functionalities provided by RDataFrame to
1832work with missing values in the dataset.
1834#### FilterAvailable and FilterMissing
1836FilterAvailable and FilterMissing are specialized RDataFrame Filter operations.
1837They take as input argument the name of a column of the dataset to watch for
1838missing values. Like Filter, they will either keep or discard an entire entry
1839based on whether a condition returns true or false. Specifically:
1841- FilterAvailable: the condition is whether the value of the column is present.
1842 If so, the entry is kept. Otherwise if the value is missing the entry is
1844- FilterMissing: the condition is whether the value of the column is missing. If
1845 so, the entry is kept. Otherwise if the value is present the entry is
1849df = ROOT.RDataFrame(dataset)
1851# Anytime an entry from "col" is missing, the entire entry will be filtered out
1852df_available = df.FilterAvailable("col")
1853df_available = df_available.Define("twice", "col * 2")
1855# Conversely, if we want to select the entries for which the column has missing
1856# values, we do the following
1857df_missingcol = df.FilterMissing("col")
1858# Following operations in the same branch of the computation graph clearly
1859# cannot access that same column, since there would be no value to read
1860df_missingcol = df_missingcol.Define("observable", "othercolumn * 2")
1864ROOT::RDataFrame df{dataset};
1866// Anytime an entry from "col" is missing, the entire entry will be filtered out
1867auto df_available = df.FilterAvailable("col");
1868auto df_twicecol = df_available.Define("twice", "col * 2");
1870// Conversely, if we want to select the entries for which the column has missing
1871// values, we do the following
1872auto df_missingcol = df.FilterMissing("col");
1873// Following operations in the same branch of the computation graph clearly
1874// cannot access that same column, since there would be no value to read
1875auto df_observable = df_missingcol.Define("observable", "othercolumn * 2");
1880DefaultValueFor creates a node of the computation graph which just forwards the
1881values of the columns necessary for other downstream nodes, when they are
1882available. In case a value of the input column passed to this function is not
1883available, the node will provide the default value passed to this function call
1887df = ROOT.RDataFrame(dataset)
1888# Anytime an entry from "col" is missing, the value will be the default one
1889default_value = ... # Some sensible default value here
1890df = df.DefaultValueFor("col", default_value)
1891df = df.Define("twice", "col * 2")
1895ROOT::RDataFrame df{dataset};
1896// Anytime an entry from "col" is missing, the value will be the default one
1897constexpr auto default_value = ... // Some sensible default value here
1898auto df_default = df.DefaultValueFor("col", default_value);
1899auto df_col = df_default.Define("twice", "col * 2");
1902#### Mixing different strategies to work with missing values in the same RDataFrame
1904All the operations presented above only act on the particular branch of the
1905computation graph where they are called, so that different results can be
1906obtained by mixing and matching the filtering or providing a default value
1910df = ROOT.RDataFrame(dataset)
1911# Anytime an entry from "col" is missing, the value will be the default one
1912default_value = ... # Some sensible default value here
1913df_default = df.DefaultValueFor("col", default_value).Define("twice", "col * 2")
1914df_filtered = df.FilterAvailable("col").Define("twice", "col * 2")
1916# Same number of total entries as the input dataset, with defaulted values
1917df_default.Display(["twice"]).Print()
1918# Only keep the entries where "col" has values
1919df_filtered.Display(["twice"]).Print()
1923ROOT::RDataFrame df{dataset};
1925// Anytime an entry from "col" is missing, the value will be the default one
1926constexpr auto default_value = ... // Some sensible default value here
1927auto df_default = df.DefaultValueFor("col", default_value).Define("twice", "col * 2");
1928auto df_filtered = df.FilterAvailable("col").Define("twice", "col * 2");
1930// Same number of total entries as the input dataset, with defaulted values
1931df_default.Display({"twice"})->Print();
1932// Only keep the entries where "col" has values
1933df_filtered.Display({"twice"})->Print();
1936#### Further considerations
1938Note that working with missing values is currently supported with a TTree-based
1939data source. Support of this functionality for other data sources may come in
1942\anchor special-values
1943### Dealing with NaN or Inf values in the dataset
1945RDataFrame does not treat NaNs or infinities beyond what the floating-point standards require, i.e. they will
1946propagate to the final result.
1947Non-finite numbers can be suppressed using Filter(), e.g.:
1950df.Filter("std::isfinite(x)").Mean("x")
1953\anchor rosetta-stone
1954### Translating TTree commands to RDataFrame
1962 <b>ROOT::RDataFrame</b>
1968// Get the tree and Draw a histogram of x for selected y values
1969auto *tree = file->Get<TTree>("myTree");
1970tree->Draw("x", "y > 2");
1975ROOT::RDataFrame df("myTree", file);
1976df.Filter("y > 2").Histo1D("x")->Draw();
1983// Draw a histogram of "jet_eta" with the desired weight
1984tree->Draw("jet_eta", "weight*(event == 1)");
1989df.Filter("event == 1").Histo1D("jet_eta", "weight")->Draw();
1996// Draw a histogram filled with values resulting from calling a method of the class of the `event` branch in the TTree.
1997tree->Draw("event.GetNtrack()");
2003df.Define("NTrack","event.GetNtrack()").Histo1D("NTrack")->Draw();
2010// Draw only every 10th event
2011tree->Draw("fNtrack","fEvtHdr.fEvtNum%10 == 0");
2016// Use the Filter operation together with the special RDF column: `rdfentry_`
2017df.Filter("rdfentry_ % 10 == 0").Histo1D("fNtrack")->Draw();
2024// object selection: for each event, fill histogram with array of selected pts
2025tree->Draw('Muon_pt', 'Muon_pt > 100');
2030// with RDF, arrays are read as ROOT::VecOps::RVec objects
2031df.Define("good_pt", "Muon_pt[Muon_pt > 100]").Histo1D("good_pt")->Draw();
2039// Draw the histogram and fill hnew with it
2040tree->Draw("sqrt(x)>>hnew","y>0");
2042// Retrieve hnew from the current directory
2043auto hnew = gDirectory->Get<TH1F>("hnew");
2048// We pass histogram constructor arguments to the Histo1D operation, to easily give the histogram a name
2049auto hist = df.Define("sqrt_x", "sqrt(x)").Filter("y>0").Histo1D({"hnew","hnew", 10, 0, 10}, "sqrt_x");
2056// Draw a 1D Profile histogram instead of TH2F
2057tree->Draw("y:x","","prof");
2059// Draw a 2D Profile histogram instead of TH3F
2060tree->Draw("z:y:x","","prof");
2066// Draw a 1D Profile histogram
2067df.Profile1D("x", "y")->Draw();
2069// Draw a 2D Profile histogram
2070df.Profile2D("x", "y", "z")->Draw();
2077// This command draws 2 entries starting with entry 5
2078tree->Draw("x", "","", 2, 5);
2083// Range function with arguments begin, end
2084df.Range(5,7).Histo1D("x")->Draw();
2091// Draw the X() component of the
2092// ROOT::Math::DisplacementVector3D in vec_list
2093tree->Draw("vec_list.X()");
2098df.Define("x", "ROOT::RVecD out; for(const auto &el: vec_list) out.push_back(el.X()); return out;").Histo1D("x")->Draw();
2105// Gather all values from a branch holding a collection per event, `pt`,
2106// and fill a histogram so that we can count the total number of values across all events
2107tree->Draw("pt>>histo");
2108auto histo = gDirectory->Get<TH1D>("histo");
2114df.Histo1D("pt")->GetEntries();
2120 <b>TTree::Scan()</b>
2123 <b>ROOT::RDataFrame</b>
2129// Print a table of the first 10 entries for all variables in the Tree
2130// if the first entry in the Muon_pt collection is > 10.
2131tree->Scan("*", "Muon_pt[0] > 10.", "", 10);
2136// Selecting columns using a regular expression
2137df.Filter("Muon_pt[0] > 10.").Display(".*", 10)->Print();
2144// For 10 events, print Muon_pt and Muon_eta, starting at entry 100
2145tree->Scan("Muon_pt:Muon_eta", "", "", 10, 100);
2150// Selecting columns using a collection of names
2151df.Range(100, 0).Display({"Muon_pt", "Muon_eta"}, 10)->Print();
2278namespace Experimental {
2332 auto *
lm = df->GetLoopManager();
2334 throw std::runtime_error(
"Cannot print information about this RDataFrame, "
2335 "it was not properly created. It must be discarded.");
2337 auto defCols =
lm->GetDefaultColumnNames();
2339 std::ostringstream
ret;
2340 if (
auto ds = df->GetDataSource()) {
2341 ret <<
"A data frame associated to the data source \"" << cling::printValue(
ds) <<
"\"";
2343 ret <<
"An empty data frame that will create " <<
lm->GetNEmptyEntries() <<
" entries\n";
Basic types used by ROOT and required by TInterpreter.
unsigned long long ULong64_t
Portable unsigned long integer 8 bytes.
ROOT::Detail::TRangeCast< T, true > TRangeDynCast
TRangeDynCast is an adapter class that allows the typed iteration through a TCollection.
The head node of a RDF computation graph.
The dataset specification for RDataFrame.
ROOT's RDataFrame offers a modern, high-level interface for analysis of data stored in TTree ,...
RDataFrame(std::string_view treeName, std::string_view filenameglob, const ColumnNames_t &defaultColumns={})
Build the dataframe.
ROOT::RDF::ColumnNames_t ColumnNames_t
Describe directory structure in memory.
A TTree represents a columnar dataset.
ROOT::RDF::Experimental::RDatasetSpec RetrieveSpecFromJson(const std::string &jsonFile)
Function to retrieve RDatasetSpec from JSON file provided.
ROOT::RDataFrame FromSpec(const std::string &jsonFile)
Factory method to create an RDataFrame from a JSON specification file.
std::vector< std::string > ColumnNames_t
std::shared_ptr< const ColumnNames_t > ColumnNamesPtr_t
