Logo ROOT  

Reference Guide
 
Loading...
Searching...
No Matches
Namespaces
TMVA_Higgs_Classification.py File Reference
Tutorials » TMVA tutorials

Detailed Description

View in nbviewer Open in SWAN
Classification example of TMVA based on public Higgs UCI dataset

The UCI data set is a public HIGGS data set , see http://archive.ics.uci.edu/ml/datasets/HIGGS used in this paper: Baldi, P., P. Sadowski, and D. Whiteson. “Searching for Exotic Particles in High-energy Physics with Deep Learning.” Nature Communications 5 (July 2, 2014).

******************************************************************************
*Tree :sig_tree : tree *
*Entries : 10000 : Total = 1177229 bytes File Size = 785298 *
* : : Tree compression factor = 1.48 *
******************************************************************************
*Br 0 :Type : Type/F *
*Entries : 10000 : Total Size= 40556 bytes File Size = 307 *
*Baskets : 1 : Basket Size= 1500672 bytes Compression= 130.54 *
*............................................................................*
*Br 1 :lepton_pT : lepton_pT/F *
*Entries : 10000 : Total Size= 40581 bytes File Size = 30464 *
*Baskets : 1 : Basket Size= 1500672 bytes Compression= 1.32 *
*............................................................................*
*Br 2 :lepton_eta : lepton_eta/F *
*Entries : 10000 : Total Size= 40586 bytes File Size = 28650 *
*Baskets : 1 : Basket Size= 1500672 bytes Compression= 1.40 *
*............................................................................*
*Br 3 :lepton_phi : lepton_phi/F *
*Entries : 10000 : Total Size= 40586 bytes File Size = 30508 *
*Baskets : 1 : Basket Size= 1500672 bytes Compression= 1.31 *
*............................................................................*
*Br 4 :missing_energy_magnitude : missing_energy_magnitude/F *
*Entries : 10000 : Total Size= 40656 bytes File Size = 35749 *
*Baskets : 1 : Basket Size= 1500672 bytes Compression= 1.12 *
*............................................................................*
*Br 5 :missing_energy_phi : missing_energy_phi/F *
*Entries : 10000 : Total Size= 40626 bytes File Size = 36766 *
*Baskets : 1 : Basket Size= 1500672 bytes Compression= 1.09 *
*............................................................................*
*Br 6 :jet1_pt : jet1_pt/F *
*Entries : 10000 : Total Size= 40571 bytes File Size = 32298 *
*Baskets : 1 : Basket Size= 1500672 bytes Compression= 1.24 *
*............................................................................*
*Br 7 :jet1_eta : jet1_eta/F *
*Entries : 10000 : Total Size= 40576 bytes File Size = 28467 *
*Baskets : 1 : Basket Size= 1500672 bytes Compression= 1.41 *
*............................................................................*
*Br 8 :jet1_phi : jet1_phi/F *
*Entries : 10000 : Total Size= 40576 bytes File Size = 30399 *
*Baskets : 1 : Basket Size= 1500672 bytes Compression= 1.32 *
*............................................................................*
*Br 9 :jet1_b-tag : jet1_b-tag/F *
*Entries : 10000 : Total Size= 40586 bytes File Size = 5087 *
*Baskets : 1 : Basket Size= 1500672 bytes Compression= 7.88 *
*............................................................................*
*Br 10 :jet2_pt : jet2_pt/F *
*Entries : 10000 : Total Size= 40571 bytes File Size = 31561 *
*Baskets : 1 : Basket Size= 1500672 bytes Compression= 1.27 *
*............................................................................*
*Br 11 :jet2_eta : jet2_eta/F *
*Entries : 10000 : Total Size= 40576 bytes File Size = 28616 *
*Baskets : 1 : Basket Size= 1500672 bytes Compression= 1.40 *
*............................................................................*
*Br 12 :jet2_phi : jet2_phi/F *
*Entries : 10000 : Total Size= 40576 bytes File Size = 30547 *
*Baskets : 1 : Basket Size= 1500672 bytes Compression= 1.31 *
*............................................................................*
*Br 13 :jet2_b-tag : jet2_b-tag/F *
*Entries : 10000 : Total Size= 40586 bytes File Size = 5031 *
*Baskets : 1 : Basket Size= 1500672 bytes Compression= 7.97 *
*............................................................................*
*Br 14 :jet3_pt : jet3_pt/F *
*Entries : 10000 : Total Size= 40571 bytes File Size = 30642 *
*Baskets : 1 : Basket Size= 1500672 bytes Compression= 1.31 *
*............................................................................*
*Br 15 :jet3_eta : jet3_eta/F *
*Entries : 10000 : Total Size= 40576 bytes File Size = 28955 *
*Baskets : 1 : Basket Size= 1500672 bytes Compression= 1.38 *
*............................................................................*
*Br 16 :jet3_phi : jet3_phi/F *
*Entries : 10000 : Total Size= 40576 bytes File Size = 30433 *
*Baskets : 1 : Basket Size= 1500672 bytes Compression= 1.32 *
*............................................................................*
*Br 17 :jet3_b-tag : jet3_b-tag/F *
*Entries : 10000 : Total Size= 40586 bytes File Size = 4879 *
*Baskets : 1 : Basket Size= 1500672 bytes Compression= 8.22 *
*............................................................................*
*Br 18 :jet4_pt : jet4_pt/F *
*Entries : 10000 : Total Size= 40571 bytes File Size = 29189 *
*Baskets : 1 : Basket Size= 1500672 bytes Compression= 1.37 *
*............................................................................*
*Br 19 :jet4_eta : jet4_eta/F *
*Entries : 10000 : Total Size= 40576 bytes File Size = 29311 *
*Baskets : 1 : Basket Size= 1500672 bytes Compression= 1.37 *
*............................................................................*
*Br 20 :jet4_phi : jet4_phi/F *
*Entries : 10000 : Total Size= 40576 bytes File Size = 30525 *
*Baskets : 1 : Basket Size= 1500672 bytes Compression= 1.31 *
*............................................................................*
*Br 21 :jet4_b-tag : jet4_b-tag/F *
*Entries : 10000 : Total Size= 40586 bytes File Size = 4725 *
*Baskets : 1 : Basket Size= 1500672 bytes Compression= 8.48 *
*............................................................................*
*Br 22 :m_jj : m_jj/F *
*Entries : 10000 : Total Size= 40556 bytes File Size = 34991 *
*Baskets : 1 : Basket Size= 1500672 bytes Compression= 1.15 *
*............................................................................*
*Br 23 :m_jjj : m_jjj/F *
*Entries : 10000 : Total Size= 40561 bytes File Size = 34460 *
*Baskets : 1 : Basket Size= 1500672 bytes Compression= 1.16 *
*............................................................................*
*Br 24 :m_lv : m_lv/F *
*Entries : 10000 : Total Size= 40556 bytes File Size = 32232 *
*Baskets : 1 : Basket Size= 1500672 bytes Compression= 1.24 *
*............................................................................*
*Br 25 :m_jlv : m_jlv/F *
*Entries : 10000 : Total Size= 40561 bytes File Size = 34598 *
*Baskets : 1 : Basket Size= 1500672 bytes Compression= 1.16 *
*............................................................................*
*Br 26 :m_bb : m_bb/F *
*Entries : 10000 : Total Size= 40556 bytes File Size = 35012 *
*Baskets : 1 : Basket Size= 1500672 bytes Compression= 1.14 *
*............................................................................*
*Br 27 :m_wbb : m_wbb/F *
*Entries : 10000 : Total Size= 40561 bytes File Size = 34493 *
*Baskets : 1 : Basket Size= 1500672 bytes Compression= 1.16 *
*............................................................................*
*Br 28 :m_wwbb : m_wwbb/F *
*Entries : 10000 : Total Size= 40566 bytes File Size = 34410 *
*Baskets : 1 : Basket Size= 1500672 bytes Compression= 1.16 *
*............................................................................*
DataSetInfo : [dataset] : Added class "Signal"
: Add Tree sig_tree of type Signal with 10000 events
DataSetInfo : [dataset] : Added class "Background"
: Add Tree bkg_tree of type Background with 10000 events
Factory : Booking method: ␛[1mLikelihood␛[0m
:
Factory : Booking method: ␛[1mFisher␛[0m
:
Factory : Booking method: ␛[1mBDT␛[0m
:
: Rebuilding Dataset dataset
: Building event vectors for type 2 Signal
: Dataset[dataset] : create input formulas for tree sig_tree
: Building event vectors for type 2 Background
: Dataset[dataset] : create input formulas for tree bkg_tree
DataSetFactory : [dataset] : Number of events in input trees
:
:
: Number of training and testing events
: ---------------------------------------------------------------------------
: Signal -- training events : 7000
: Signal -- testing events : 3000
: Signal -- training and testing events: 10000
: Background -- training events : 7000
: Background -- testing events : 3000
: Background -- training and testing events: 10000
:
DataSetInfo : Correlation matrix (Signal):
: ----------------------------------------------------------------
: m_jj m_jjj m_lv m_jlv m_bb m_wbb m_wwbb
: m_jj: +1.000 +0.777 +0.010 +0.107 +0.036 +0.517 +0.532
: m_jjj: +0.777 +1.000 +0.006 +0.083 +0.157 +0.682 +0.669
: m_lv: +0.010 +0.006 +1.000 +0.111 -0.026 +0.011 +0.023
: m_jlv: +0.107 +0.083 +0.111 +1.000 +0.325 +0.550 +0.555
: m_bb: +0.036 +0.157 -0.026 +0.325 +1.000 +0.463 +0.347
: m_wbb: +0.517 +0.682 +0.011 +0.550 +0.463 +1.000 +0.912
: m_wwbb: +0.532 +0.669 +0.023 +0.555 +0.347 +0.912 +1.000
: ----------------------------------------------------------------
DataSetInfo : Correlation matrix (Background):
: ----------------------------------------------------------------
: m_jj m_jjj m_lv m_jlv m_bb m_wbb m_wwbb
: m_jj: +1.000 +0.804 +0.017 +0.125 +0.007 +0.381 +0.394
: m_jjj: +0.804 +1.000 +0.025 +0.159 +0.153 +0.535 +0.520
: m_lv: +0.017 +0.025 +1.000 +0.114 +0.042 +0.064 +0.069
: m_jlv: +0.125 +0.159 +0.114 +1.000 +0.286 +0.592 +0.542
: m_bb: +0.007 +0.153 +0.042 +0.286 +1.000 +0.623 +0.441
: m_wbb: +0.381 +0.535 +0.064 +0.592 +0.623 +1.000 +0.878
: m_wwbb: +0.394 +0.520 +0.069 +0.542 +0.441 +0.878 +1.000
: ----------------------------------------------------------------
DataSetFactory : [dataset] :
:
Factory : Booking method: ␛[1mDNN_CPU␛[0m
:
: Parsing option string:
: ... "!H:V:ErrorStrategy=CROSSENTROPY:VarTransform=G:WeightInitialization=XAVIER:InputLayout=1|1|7:BatchLayout=1|128|7:Layout=DENSE|64|TANH,DENSE|64|TANH,DENSE|64|TANH,DENSE|64|TANH,DENSE|1|LINEAR:TrainingStrategy=LearningRate=1e-3,Momentum=0.9,ConvergenceSteps=10,BatchSize=128,TestRepetitions=1,MaxEpochs=20,WeightDecay=1e-4,Regularization=None,Optimizer=ADAM,ADAM_beta1=0.9,ADAM_beta2=0.999,ADAM_eps=1.E-7,DropConfig=0.0+0.0+0.0+0.:Architecture=CPU"
: The following options are set:
: - By User:
: <none>
: - Default:
: Boost_num: "0" [Number of times the classifier will be boosted]
: Parsing option string:
: ... "!H:V:ErrorStrategy=CROSSENTROPY:VarTransform=G:WeightInitialization=XAVIER:InputLayout=1|1|7:BatchLayout=1|128|7:Layout=DENSE|64|TANH,DENSE|64|TANH,DENSE|64|TANH,DENSE|64|TANH,DENSE|1|LINEAR:TrainingStrategy=LearningRate=1e-3,Momentum=0.9,ConvergenceSteps=10,BatchSize=128,TestRepetitions=1,MaxEpochs=20,WeightDecay=1e-4,Regularization=None,Optimizer=ADAM,ADAM_beta1=0.9,ADAM_beta2=0.999,ADAM_eps=1.E-7,DropConfig=0.0+0.0+0.0+0.:Architecture=CPU"
: The following options are set:
: - By User:
: V: "True" [Verbose output (short form of "VerbosityLevel" below - overrides the latter one)]
: VarTransform: "G" [List of variable transformations performed before training, e.g., "D_Background,P_Signal,G,N_AllClasses" for: "Decorrelation, PCA-transformation, Gaussianisation, Normalisation, each for the given class of events ('AllClasses' denotes all events of all classes, if no class indication is given, 'All' is assumed)"]
: H: "False" [Print method-specific help message]
: InputLayout: "1|1|7" [The Layout of the input]
: BatchLayout: "1|128|7" [The Layout of the batch]
: Layout: "DENSE|64|TANH,DENSE|64|TANH,DENSE|64|TANH,DENSE|64|TANH,DENSE|1|LINEAR" [Layout of the network.]
: ErrorStrategy: "CROSSENTROPY" [Loss function: Mean squared error (regression) or cross entropy (binary classification).]
: WeightInitialization: "XAVIER" [Weight initialization strategy]
: Architecture: "CPU" [Which architecture to perform the training on.]
: TrainingStrategy: "LearningRate=1e-3,Momentum=0.9,ConvergenceSteps=10,BatchSize=128,TestRepetitions=1,MaxEpochs=20,WeightDecay=1e-4,Regularization=None,Optimizer=ADAM,ADAM_beta1=0.9,ADAM_beta2=0.999,ADAM_eps=1.E-7,DropConfig=0.0+0.0+0.0+0." [Defines the training strategies.]
: - Default:
: VerbosityLevel: "Default" [Verbosity level]
: CreateMVAPdfs: "False" [Create PDFs for classifier outputs (signal and background)]
: IgnoreNegWeightsInTraining: "False" [Events with negative weights are ignored in the training (but are included for testing and performance evaluation)]
: RandomSeed: "0" [Random seed used for weight initialization and batch shuffling]
: ValidationSize: "20%" [Part of the training data to use for validation. Specify as 0.2 or 20% to use a fifth of the data set as validation set. Specify as 100 to use exactly 100 events. (Default: 20%)]
DNN_CPU : [dataset] : Create Transformation "G" with events from all classes.
:
: Transformation, Variable selection :
: Input : variable 'm_jj' <---> Output : variable 'm_jj'
: Input : variable 'm_jjj' <---> Output : variable 'm_jjj'
: Input : variable 'm_lv' <---> Output : variable 'm_lv'
: Input : variable 'm_jlv' <---> Output : variable 'm_jlv'
: Input : variable 'm_bb' <---> Output : variable 'm_bb'
: Input : variable 'm_wbb' <---> Output : variable 'm_wbb'
: Input : variable 'm_wwbb' <---> Output : variable 'm_wwbb'
: Will now use the CPU architecture with BLAS and IMT support !
Factory : ␛[1mTrain all methods␛[0m
Factory : [dataset] : Create Transformation "I" with events from all classes.
:
: Transformation, Variable selection :
: Input : variable 'm_jj' <---> Output : variable 'm_jj'
: Input : variable 'm_jjj' <---> Output : variable 'm_jjj'
: Input : variable 'm_lv' <---> Output : variable 'm_lv'
: Input : variable 'm_jlv' <---> Output : variable 'm_jlv'
: Input : variable 'm_bb' <---> Output : variable 'm_bb'
: Input : variable 'm_wbb' <---> Output : variable 'm_wbb'
: Input : variable 'm_wwbb' <---> Output : variable 'm_wwbb'
TFHandler_Factory : Variable Mean RMS [ Min Max ]
: -----------------------------------------------------------
: m_jj: 1.0352 0.65399 [ 0.14661 13.098 ]
: m_jjj: 1.0218 0.36964 [ 0.34201 7.3920 ]
: m_lv: 1.0497 0.16065 [ 0.26679 3.6823 ]
: m_jlv: 1.0126 0.39935 [ 0.38441 6.5831 ]
: m_bb: 0.98070 0.53223 [ 0.093482 7.8598 ]
: m_wbb: 1.0338 0.35968 [ 0.38503 4.5425 ]
: m_wwbb: 0.96049 0.31009 [ 0.43228 4.0728 ]
: -----------------------------------------------------------
: Ranking input variables (method unspecific)...
IdTransformation : Ranking result (top variable is best ranked)
: -------------------------------
: Rank : Variable : Separation
: -------------------------------
: 1 : m_bb : 9.114e-02
: 2 : m_wwbb : 4.330e-02
: 3 : m_wbb : 4.241e-02
: 4 : m_jjj : 2.875e-02
: 5 : m_jlv : 1.905e-02
: 6 : m_jj : 3.432e-03
: 7 : m_lv : 2.855e-03
: -------------------------------
Factory : Train method: Likelihood for Classification
:
:
: ␛[1m================================================================␛[0m
: ␛[1mH e l p f o r M V A m e t h o d [ Likelihood ] :␛[0m
:
: ␛[1m--- Short description:␛[0m
:
: The maximum-likelihood classifier models the data with probability
: density functions (PDF) reproducing the signal and background
: distributions of the input variables. Correlations among the
: variables are ignored.
:
: ␛[1m--- Performance optimisation:␛[0m
:
: Required for good performance are decorrelated input variables
: (PCA transformation via the option "VarTransform=Decorrelate"
: may be tried). Irreducible non-linear correlations may be reduced
: by precombining strongly correlated input variables, or by simply
: removing one of the variables.
:
: ␛[1m--- Performance tuning via configuration options:␛[0m
:
: High fidelity PDF estimates are mandatory, i.e., sufficient training
: statistics is required to populate the tails of the distributions
: It would be a surprise if the default Spline or KDE kernel parameters
: provide a satisfying fit to the data. The user is advised to properly
: tune the events per bin and smooth options in the spline cases
: individually per variable. If the KDE kernel is used, the adaptive
: Gaussian kernel may lead to artefacts, so please always also try
: the non-adaptive one.
:
: All tuning parameters must be adjusted individually for each input
: variable!
:
: <Suppress this message by specifying "!H" in the booking option>
: ␛[1m================================================================␛[0m
:
: Filling reference histograms
: Building PDF out of reference histograms
: Elapsed time for training with 14000 events: 0.0804 sec
Likelihood : [dataset] : Evaluation of Likelihood on training sample (14000 events)
: Elapsed time for evaluation of 14000 events: 0.0109 sec
: Creating xml weight file: ␛[0;36mdataset/weights/TMVA_Higgs_Classification_Likelihood.weights.xml␛[0m
: Creating standalone class: ␛[0;36mdataset/weights/TMVA_Higgs_Classification_Likelihood.class.C␛[0m
: Higgs_ClassificationOutput.root:/dataset/Method_Likelihood/Likelihood
Factory : Training finished
:
Factory : Train method: Fisher for Classification
:
:
: ␛[1m================================================================␛[0m
: ␛[1mH e l p f o r M V A m e t h o d [ Fisher ] :␛[0m
:
: ␛[1m--- Short description:␛[0m
:
: Fisher discriminants select events by distinguishing the mean
: values of the signal and background distributions in a trans-
: formed variable space where linear correlations are removed.
:
: (More precisely: the "linear discriminator" determines
: an axis in the (correlated) hyperspace of the input
: variables such that, when projecting the output classes
: (signal and background) upon this axis, they are pushed
: as far as possible away from each other, while events
: of a same class are confined in a close vicinity. The
: linearity property of this classifier is reflected in the
: metric with which "far apart" and "close vicinity" are
: determined: the covariance matrix of the discriminating
: variable space.)
:
: ␛[1m--- Performance optimisation:␛[0m
:
: Optimal performance for Fisher discriminants is obtained for
: linearly correlated Gaussian-distributed variables. Any deviation
: from this ideal reduces the achievable separation power. In
: particular, no discrimination at all is achieved for a variable
: that has the same sample mean for signal and background, even if
: the shapes of the distributions are very different. Thus, Fisher
: discriminants often benefit from suitable transformations of the
: input variables. For example, if a variable x in [-1,1] has a
: a parabolic signal distributions, and a uniform background
: distributions, their mean value is zero in both cases, leading
: to no separation. The simple transformation x -> |x| renders this
: variable powerful for the use in a Fisher discriminant.
:
: ␛[1m--- Performance tuning via configuration options:␛[0m
:
: <None>
:
: <Suppress this message by specifying "!H" in the booking option>
: ␛[1m================================================================␛[0m
:
Fisher : Results for Fisher coefficients:
: -----------------------
: Variable: Coefficient:
: -----------------------
: m_jj: -0.051
: m_jjj: +0.187
: m_lv: +0.037
: m_jlv: +0.065
: m_bb: -0.207
: m_wbb: +0.532
: m_wwbb: -0.743
: (offset): +0.125
: -----------------------
: Elapsed time for training with 14000 events: 0.00605 sec
Fisher : [dataset] : Evaluation of Fisher on training sample (14000 events)
: Elapsed time for evaluation of 14000 events: 0.00098 sec
: <CreateMVAPdfs> Separation from histogram (PDF): 0.085 (0.000)
: Dataset[dataset] : Evaluation of Fisher on training sample
: Creating xml weight file: ␛[0;36mdataset/weights/TMVA_Higgs_Classification_Fisher.weights.xml␛[0m
: Creating standalone class: ␛[0;36mdataset/weights/TMVA_Higgs_Classification_Fisher.class.C␛[0m
Factory : Training finished
:
Factory : Train method: BDT for Classification
:
BDT : #events: (reweighted) sig: 7000 bkg: 7000
: #events: (unweighted) sig: 7000 bkg: 7000
: Training 200 Decision Trees ... patience please
: Elapsed time for training with 14000 events: 0.525 sec
BDT : [dataset] : Evaluation of BDT on training sample (14000 events)
: Elapsed time for evaluation of 14000 events: 0.0564 sec
: Creating xml weight file: ␛[0;36mdataset/weights/TMVA_Higgs_Classification_BDT.weights.xml␛[0m
: Creating standalone class: ␛[0;36mdataset/weights/TMVA_Higgs_Classification_BDT.class.C␛[0m
: Higgs_ClassificationOutput.root:/dataset/Method_BDT/BDT
Factory : Training finished
:
Factory : Train method: DNN_CPU for Classification
:
: Preparing the Gaussian transformation...
TFHandler_DNN_CPU : Variable Mean RMS [ Min Max ]
: -----------------------------------------------------------
: m_jj: 0.0042139 0.99787 [ -3.2801 5.7307 ]
: m_jjj: 0.0043508 0.99784 [ -3.2805 5.7307 ]
: m_lv: 0.0051611 1.0008 [ -3.2813 5.7307 ]
: m_jlv: 0.0044388 0.99830 [ -3.2803 5.7307 ]
: m_bb: 0.0041864 0.99765 [ -3.2793 5.7307 ]
: m_wbb: 0.0046426 0.99950 [ -3.2802 5.7307 ]
: m_wwbb: 0.0044594 0.99873 [ -3.2802 5.7307 ]
: -----------------------------------------------------------
: Start of deep neural network training on CPU using MT, nthreads = 1
:
TFHandler_DNN_CPU : Variable Mean RMS [ Min Max ]
: -----------------------------------------------------------
: m_jj: 0.0042139 0.99787 [ -3.2801 5.7307 ]
: m_jjj: 0.0043508 0.99784 [ -3.2805 5.7307 ]
: m_lv: 0.0051611 1.0008 [ -3.2813 5.7307 ]
: m_jlv: 0.0044388 0.99830 [ -3.2803 5.7307 ]
: m_bb: 0.0041864 0.99765 [ -3.2793 5.7307 ]
: m_wbb: 0.0046426 0.99950 [ -3.2802 5.7307 ]
: m_wwbb: 0.0044594 0.99873 [ -3.2802 5.7307 ]
: -----------------------------------------------------------
: ***** Deep Learning Network *****
DEEP NEURAL NETWORK: Depth = 5 Input = ( 1, 1, 7 ) Batch size = 128 Loss function = C
Layer 0 DENSE Layer: ( Input = 7 , Width = 64 ) Output = ( 1 , 128 , 64 ) Activation Function = Tanh
Layer 1 DENSE Layer: ( Input = 64 , Width = 64 ) Output = ( 1 , 128 , 64 ) Activation Function = Tanh
Layer 2 DENSE Layer: ( Input = 64 , Width = 64 ) Output = ( 1 , 128 , 64 ) Activation Function = Tanh
Layer 3 DENSE Layer: ( Input = 64 , Width = 64 ) Output = ( 1 , 128 , 64 ) Activation Function = Tanh
Layer 4 DENSE Layer: ( Input = 64 , Width = 1 ) Output = ( 1 , 128 , 1 ) Activation Function = Identity
: Using 11200 events for training and 2800 for testing
: Compute initial loss on the validation data
: Training phase 1 of 1: Optimizer ADAM (beta1=0.9,beta2=0.999,eps=1e-07) Learning rate = 0.001 regularization 0 minimum error = 0.842379
: --------------------------------------------------------------
: Epoch | Train Err. Val. Err. t(s)/epoch t(s)/Loss nEvents/s Conv. Steps
: --------------------------------------------------------------
: Start epoch iteration ...
: 1 Minimum Test error found - save the configuration
: 1 | 0.656312 0.628024 0.136013 0.0134859 90886.2 0
: 2 Minimum Test error found - save the configuration
: 2 | 0.610319 0.594992 0.137034 0.0138206 90380 0
: 3 Minimum Test error found - save the configuration
: 3 | 0.587754 0.587918 0.137008 0.013656 90278 0
: 4 Minimum Test error found - save the configuration
: 4 | 0.580804 0.580711 0.137891 0.0145341 90274.4 0
: 5 | 0.573675 0.583369 0.136951 0.0137188 90365.8 1
: 6 Minimum Test error found - save the configuration
: 6 | 0.570347 0.578231 0.140281 0.0140071 88189.5 0
: 7 Minimum Test error found - save the configuration
: 7 | 0.56428 0.577981 0.138729 0.0138804 89196 0
: 8 Minimum Test error found - save the configuration
: 8 | 0.560366 0.573655 0.137929 0.0137141 89650.9 0
: 9 | 0.559089 0.574563 0.136457 0.0134217 90510.9 1
: 10 | 0.557409 0.577052 0.138816 0.015425 90249.5 2
: 11 | 0.556713 0.577258 0.138611 0.013812 89231.7 3
: 12 | 0.552378 0.574036 0.137798 0.0137546 89774.9 4
: 13 | 0.553087 0.581924 0.137596 0.0135532 89775.5 5
: 14 Minimum Test error found - save the configuration
: 14 | 0.551422 0.573479 0.135548 0.0135428 91274.7 0
: 15 | 0.548133 0.58033 0.135527 0.0137539 91448.4 1
: 16 | 0.547583 0.578732 0.138586 0.0135347 89051.2 2
: 17 | 0.548444 0.578202 0.136314 0.0137248 90840.1 3
: 18 | 0.544113 0.578944 0.137448 0.014024 90225.3 4
: 19 | 0.545359 0.586514 0.141126 0.0137395 87418.9 5
: 20 | 0.542031 0.582552 0.137173 0.0136212 90132.3 6
:
: Elapsed time for training with 14000 events: 2.8 sec
: Evaluate deep neural network on CPU using batches with size = 128
:
DNN_CPU : [dataset] : Evaluation of DNN_CPU on training sample (14000 events)
: Elapsed time for evaluation of 14000 events: 0.0711 sec
: Creating xml weight file: ␛[0;36mdataset/weights/TMVA_Higgs_Classification_DNN_CPU.weights.xml␛[0m
: Creating standalone class: ␛[0;36mdataset/weights/TMVA_Higgs_Classification_DNN_CPU.class.C␛[0m
Factory : Training finished
:
: Ranking input variables (method specific)...
Likelihood : Ranking result (top variable is best ranked)
: -------------------------------------
: Rank : Variable : Delta Separation
: -------------------------------------
: 1 : m_bb : 5.483e-02
: 2 : m_wbb : 4.416e-02
: 3 : m_wwbb : 4.201e-02
: 4 : m_jjj : 4.949e-03
: 5 : m_lv : -2.642e-03
: 6 : m_jj : -5.516e-03
: 7 : m_jlv : -1.042e-02
: -------------------------------------
Fisher : Ranking result (top variable is best ranked)
: ---------------------------------
: Rank : Variable : Discr. power
: ---------------------------------
: 1 : m_bb : 1.180e-02
: 2 : m_wwbb : 7.816e-03
: 3 : m_wbb : 2.085e-03
: 4 : m_jlv : 5.619e-04
: 5 : m_jjj : 2.327e-04
: 6 : m_lv : 3.319e-05
: 7 : m_jj : 1.479e-05
: ---------------------------------
BDT : Ranking result (top variable is best ranked)
: ----------------------------------------
: Rank : Variable : Variable Importance
: ----------------------------------------
: 1 : m_bb : 2.045e-01
: 2 : m_wwbb : 1.687e-01
: 3 : m_jlv : 1.638e-01
: 4 : m_jjj : 1.413e-01
: 5 : m_wbb : 1.356e-01
: 6 : m_jj : 1.080e-01
: 7 : m_lv : 7.813e-02
: ----------------------------------------
: No variable ranking supplied by classifier: DNN_CPU
TH1.Print Name = TrainingHistory_DNN_CPU_trainingError, Entries= 0, Total sum= 11.3096
TH1.Print Name = TrainingHistory_DNN_CPU_valError, Entries= 0, Total sum= 11.6485
Factory : === Destroy and recreate all methods via weight files for testing ===
:
: Reading weight file: ␛[0;36mdataset/weights/TMVA_Higgs_Classification_Likelihood.weights.xml␛[0m
: Reading weight file: ␛[0;36mdataset/weights/TMVA_Higgs_Classification_Fisher.weights.xml␛[0m
: Reading weight file: ␛[0;36mdataset/weights/TMVA_Higgs_Classification_BDT.weights.xml␛[0m
: Reading weight file: ␛[0;36mdataset/weights/TMVA_Higgs_Classification_DNN_CPU.weights.xml␛[0m
Factory : ␛[1mTest all methods␛[0m
Factory : Test method: Likelihood for Classification performance
:
Likelihood : [dataset] : Evaluation of Likelihood on testing sample (6000 events)
: Elapsed time for evaluation of 6000 events: 0.00564 sec
Factory : Test method: Fisher for Classification performance
:
Fisher : [dataset] : Evaluation of Fisher on testing sample (6000 events)
: Elapsed time for evaluation of 6000 events: 0.00164 sec
: Dataset[dataset] : Evaluation of Fisher on testing sample
Factory : Test method: BDT for Classification performance
:
BDT : [dataset] : Evaluation of BDT on testing sample (6000 events)
: Elapsed time for evaluation of 6000 events: 0.0207 sec
Factory : Test method: DNN_CPU for Classification performance
:
: Evaluate deep neural network on CPU using batches with size = 1000
:
TFHandler_DNN_CPU : Variable Mean RMS [ Min Max ]
: -----------------------------------------------------------
: m_jj: 0.029995 0.98065 [ -3.1064 5.7307 ]
: m_jjj: 0.030151 0.98464 [ -2.9982 5.7307 ]
: m_lv: 0.011982 1.0066 [ -3.2274 5.7307 ]
: m_jlv: 0.0049774 1.0015 [ -3.0644 5.7307 ]
: m_bb: -0.036143 1.0111 [ -5.7307 5.7307 ]
: m_wbb: -0.0056377 1.0239 [ -3.0260 5.7307 ]
: m_wwbb: 0.0023364 1.0091 [ -3.1905 5.7307 ]
: -----------------------------------------------------------
DNN_CPU : [dataset] : Evaluation of DNN_CPU on testing sample (6000 events)
: Elapsed time for evaluation of 6000 events: 0.0299 sec
Factory : ␛[1mEvaluate all methods␛[0m
Factory : Evaluate classifier: Likelihood
:
Likelihood : [dataset] : Loop over test events and fill histograms with classifier response...
:
TFHandler_Likelihood : Variable Mean RMS [ Min Max ]
: -----------------------------------------------------------
: m_jj: 1.0368 0.66752 [ 0.16310 16.132 ]
: m_jjj: 1.0272 0.38070 [ 0.41899 8.9401 ]
: m_lv: 1.0522 0.17017 [ 0.29757 3.2605 ]
: m_jlv: 1.0135 0.40315 [ 0.41660 5.8195 ]
: m_bb: 0.96616 0.53867 [ 0.080986 8.2551 ]
: m_wbb: 1.0344 0.37776 [ 0.42068 6.4013 ]
: m_wwbb: 0.96122 0.31782 [ 0.44118 4.5350 ]
: -----------------------------------------------------------
Factory : Evaluate classifier: Fisher
:
Fisher : [dataset] : Loop over test events and fill histograms with classifier response...
:
: Also filling probability and rarity histograms (on request)...
TFHandler_Fisher : Variable Mean RMS [ Min Max ]
: -----------------------------------------------------------
: m_jj: 1.0368 0.66752 [ 0.16310 16.132 ]
: m_jjj: 1.0272 0.38070 [ 0.41899 8.9401 ]
: m_lv: 1.0522 0.17017 [ 0.29757 3.2605 ]
: m_jlv: 1.0135 0.40315 [ 0.41660 5.8195 ]
: m_bb: 0.96616 0.53867 [ 0.080986 8.2551 ]
: m_wbb: 1.0344 0.37776 [ 0.42068 6.4013 ]
: m_wwbb: 0.96122 0.31782 [ 0.44118 4.5350 ]
: -----------------------------------------------------------
Factory : Evaluate classifier: BDT
:
BDT : [dataset] : Loop over test events and fill histograms with classifier response...
:
TFHandler_BDT : Variable Mean RMS [ Min Max ]
: -----------------------------------------------------------
: m_jj: 1.0368 0.66752 [ 0.16310 16.132 ]
: m_jjj: 1.0272 0.38070 [ 0.41899 8.9401 ]
: m_lv: 1.0522 0.17017 [ 0.29757 3.2605 ]
: m_jlv: 1.0135 0.40315 [ 0.41660 5.8195 ]
: m_bb: 0.96616 0.53867 [ 0.080986 8.2551 ]
: m_wbb: 1.0344 0.37776 [ 0.42068 6.4013 ]
: m_wwbb: 0.96122 0.31782 [ 0.44118 4.5350 ]
: -----------------------------------------------------------
Factory : Evaluate classifier: DNN_CPU
:
DNN_CPU : [dataset] : Loop over test events and fill histograms with classifier response...
:
: Evaluate deep neural network on CPU using batches with size = 1000
:
TFHandler_DNN_CPU : Variable Mean RMS [ Min Max ]
: -----------------------------------------------------------
: m_jj: 0.0042139 0.99787 [ -3.2801 5.7307 ]
: m_jjj: 0.0043508 0.99784 [ -3.2805 5.7307 ]
: m_lv: 0.0051611 1.0008 [ -3.2813 5.7307 ]
: m_jlv: 0.0044388 0.99830 [ -3.2803 5.7307 ]
: m_bb: 0.0041864 0.99765 [ -3.2793 5.7307 ]
: m_wbb: 0.0046426 0.99950 [ -3.2802 5.7307 ]
: m_wwbb: 0.0044594 0.99873 [ -3.2802 5.7307 ]
: -----------------------------------------------------------
TFHandler_DNN_CPU : Variable Mean RMS [ Min Max ]
: -----------------------------------------------------------
: m_jj: 0.029995 0.98065 [ -3.1064 5.7307 ]
: m_jjj: 0.030151 0.98464 [ -2.9982 5.7307 ]
: m_lv: 0.011982 1.0066 [ -3.2274 5.7307 ]
: m_jlv: 0.0049774 1.0015 [ -3.0644 5.7307 ]
: m_bb: -0.036143 1.0111 [ -5.7307 5.7307 ]
: m_wbb: -0.0056377 1.0239 [ -3.0260 5.7307 ]
: m_wwbb: 0.0023364 1.0091 [ -3.1905 5.7307 ]
: -----------------------------------------------------------
:
: Evaluation results ranked by best signal efficiency and purity (area)
: -------------------------------------------------------------------------------------------------------------------
: DataSet MVA
: Name: Method: ROC-integ
: dataset DNN_CPU : 0.768
: dataset BDT : 0.758
: dataset Likelihood : 0.699
: dataset Fisher : 0.654
: -------------------------------------------------------------------------------------------------------------------
:
: Testing efficiency compared to training efficiency (overtraining check)
: -------------------------------------------------------------------------------------------------------------------
: DataSet MVA Signal efficiency: from test sample (from training sample)
: Name: Method: @B=0.01 @B=0.10 @B=0.30
: -------------------------------------------------------------------------------------------------------------------
: dataset DNN_CPU : 0.112 (0.144) 0.420 (0.449) 0.681 (0.706)
: dataset BDT : 0.080 (0.095) 0.394 (0.394) 0.674 (0.685)
: dataset Likelihood : 0.058 (0.076) 0.309 (0.338) 0.584 (0.590)
: dataset Fisher : 0.017 (0.014) 0.128 (0.141) 0.500 (0.529)
: -------------------------------------------------------------------------------------------------------------------
:
Dataset:dataset : Created tree 'TestTree' with 6000 events
:
Dataset:dataset : Created tree 'TrainTree' with 14000 events
:
Factory : ␛[1mThank you for using TMVA!␛[0m
: ␛[1mFor citation information, please visit: http://tmva.sf.net/citeTMVA.html␛[0m
## Declare Factory
## Create the Factory class. Later you can choose the methods
## whose performance you'd like to investigate.
## The factory is the major TMVA object you have to interact with. Here is the list of parameters you need to pass
## - The first argument is the base of the name of all the output
## weightfiles in the directory weight/ that will be created with the
## method parameters
## - The second argument is the output file for the training results
## - The third argument is a string option defining some general configuration for the TMVA session. For example all TMVA output can be suppressed by removing the "!" (not) in front of the "Silent" argument in the option string
import ROOT
import os
TMVA = ROOT.TMVA
TFile = ROOT.TFile
TMVA.Tools.Instance()
# options to control used methods
useLikelihood = True # likelihood based discriminant
useLikelihoodKDE = False # likelihood based discriminant
useFischer = True # Fischer discriminant
useMLP = False # Multi Layer Perceptron (old TMVA NN implementation)
useBDT = True # Boosted Decision Tree
useDL = True # TMVA Deep learning ( CPU or GPU)
useKeras = True # Use Keras Deep Learning via PyMVA
if ROOT.gSystem.GetFromPipe("root-config --has-tmva-pymva") == "yes":
TMVA.PyMethodBase.PyInitialize()
else:
useKeras = False # cannot use Keras if PYMVA is not available
if useKeras:
try:
import tensorflow
except:
ROOT.Warning("TMVA_Higgs_Classification", "Skip using Keras since tensorflow is not available")
useKeras = False
outputFile = TFile.Open("Higgs_ClassificationOutput.root", "RECREATE")
factory = TMVA.Factory(
"TMVA_Higgs_Classification", outputFile, V=False, ROC=True, Silent=False, Color=True, AnalysisType="Classification"
)
## Setup Dataset(s)
# Define now input data file and signal and background trees
inputFileName = str(ROOT.gROOT.GetTutorialDir()) + "/tmva/data/Higgs_data.root"
inputFile = TFile.Open(inputFileName)
if inputFile is None:
raise FileNotFoundError("Input file is not found - exit")
# --- Register the training and test trees
signalTree = inputFile.Get("sig_tree")
backgroundTree = inputFile.Get("bkg_tree")
signalTree.Print()
## Declare DataLoader(s)
# The next step is to declare the DataLoader class that deals with input variables
# Define the input variables that shall be used for the MVA training
# note that you may also use variable expressions, which can be parsed by TTree::Draw( "expression" )]
loader = TMVA.DataLoader("dataset")
loader.AddVariable("m_jj")
loader.AddVariable("m_jjj")
loader.AddVariable("m_lv")
loader.AddVariable("m_jlv")
loader.AddVariable("m_bb")
loader.AddVariable("m_wbb")
loader.AddVariable("m_wwbb")
# We set now the input data trees in the TMVA DataLoader class
# global event weights per tree (see below for setting event-wise weights)
signalWeight = 1.0
backgroundWeight = 1.0
# You can add an arbitrary number of signal or background trees
loader.AddSignalTree(signalTree, signalWeight)
loader.AddBackgroundTree(backgroundTree, backgroundWeight)
# Set individual event weights (the variables must exist in the original TTree)
# for signal : factory->SetSignalWeightExpression ("weight1*weight2");
# for background: factory->SetBackgroundWeightExpression("weight1*weight2");
# loader->SetBackgroundWeightExpression( "weight" );
# Apply additional cuts on the signal and background samples (can be different)
mycuts = ROOT.TCut("") # for example: TCut mycuts = "abs(var1)<0.5 && abs(var2-0.5)<1";
mycutb = ROOT.TCut("") # for example: TCut mycutb = "abs(var1)<0.5";
# Tell the factory how to use the training and testing events
#
# If no numbers of events are given, half of the events in the tree are used
# for training, and the other half for testing:
# loader->PrepareTrainingAndTestTree( mycut, "SplitMode=random:!V" );
# To also specify the number of testing events, use:
loader.PrepareTrainingAndTestTree(
mycuts, mycutb, nTrain_Signal=7000, nTrain_Background=7000, SplitMode="Random", NormMode="NumEvents", V=False
)
## Booking Methods
# Here we book the TMVA methods. We book first a Likelihood based on KDE (Kernel Density Estimation), a Fischer discriminant, a BDT
# and a shallow neural network
# Likelihood ("naive Bayes estimator")
if useLikelihood:
factory.BookMethod(
loader,
TMVA.Types.kLikelihood,
"Likelihood",
H=True,
V=False,
TransformOutput=True,
PDFInterpol="Spline2:NSmoothSig[0]=20:NSmoothBkg[0]=20:NSmoothBkg[1]=10",
NSmooth=1,
NAvEvtPerBin=50,
)
# Use a kernel density estimator to approximate the PDFs
if useLikelihoodKDE:
factory.BookMethod(
loader,
TMVA.Types.kLikelihood,
"LikelihoodKDE",
H=False,
V=False,
TransformOutput=False,
PDFInterpol="KDE",
KDEtype="Gauss",
KDEiter="Adaptive",
KDEFineFactor=0.3,
KDEborder=None,
NAvEvtPerBin=50,
)
# Fisher discriminant (same as LD)
if useFischer:
factory.BookMethod(
loader,
TMVA.Types.kFisher,
"Fisher",
H=True,
V=False,
Fisher=True,
VarTransform=None,
CreateMVAPdfs=True,
PDFInterpolMVAPdf="Spline2",
NbinsMVAPdf=50,
NsmoothMVAPdf=10,
)
# Boosted Decision Trees
if useBDT:
factory.BookMethod(
loader,
TMVA.Types.kBDT,
"BDT",
V=False,
NTrees=200,
MinNodeSize="2.5%",
MaxDepth=2,
BoostType="AdaBoost",
AdaBoostBeta=0.5,
UseBaggedBoost=True,
BaggedSampleFraction=0.5,
SeparationType="GiniIndex",
nCuts=20,
)
# Multi-Layer Perceptron (Neural Network)
if useMLP:
factory.BookMethod(
loader,
TMVA.Types.kMLP,
"MLP",
H=False,
V=False,
NeuronType="tanh",
VarTransform="N",
NCycles=100,
HiddenLayers="N+5",
TestRate=5,
UseRegulator=False,
)
## Here we book the new DNN of TMVA if we have support in ROOT. We will use GPU version if ROOT is enabled with GPU
## Booking Deep Neural Network
# Here we define the option string for building the Deep Neural network model.
#### 1. Define DNN layout
# The DNN configuration is defined using a string. Note that whitespaces between characters are not allowed.
# We define first the DNN layout:
# - **input layout** : this defines the input data format for the DNN as ``input depth | height | width``.
# In case of a dense layer as first layer the input layout should be ``1 | 1 | number of input variables`` (features)
# - **batch layout** : this defines how are the input batch. It is related to input layout but not the same.
# If the first layer is dense it should be ``1 | batch size ! number of variables`` (features)
# *(note the use of the character `|` as separator of input parameters for DNN layout)*
# note that in case of only dense layer the input layout could be omitted but it is required when defining more
# complex architectures
# - **layer layout** string defining the layer architecture. The syntax is
# - layer type (e.g. DENSE, CONV, RNN)
# - layer parameters (e.g. number of units)
# - activation function (e.g TANH, RELU,...)
# *the different layers are separated by the ``","`` *
#### 2. Define Training Strategy
# We define here the training strategy parameters for the DNN. The parameters are separated by the ``","`` separator.
# One can then concatenate different training strategy with different parameters. The training strategy are separated by
# the ``"|"`` separator.
# - Optimizer
# - Learning rate
# - Momentum (valid for SGD and RMSPROP)
# - Regularization and Weight Decay
# - Dropout
# - Max number of epochs
# - Convergence steps. if the test error will not decrease after that value the training will stop
# - Batch size (This value must be the same specified in the input layout)
# - Test Repetitions (the interval when the test error will be computed)
#### 3. Define general DNN options
# We define the general DNN options concatenating in the final string the previously defined layout and training strategy.
# Note we use the ``":"`` separator to separate the different higher level options, as in the other TMVA methods.
# In addition to input layout, batch layout and training strategy we add now:
# - Type of Loss function (e.g. CROSSENTROPY)
# - Weight Initizalization (e.g XAVIER, XAVIERUNIFORM, NORMAL )
# - Variable Transformation
# - Type of Architecture (e.g. CPU, GPU, Standard)
# We can then book the DL method using the built option string
if useDL:
useDLGPU = ROOT.gSystem.GetFromPipe("root-config --has-tmva-gpu") == "yes"
# Define DNN layout
# Define Training strategies
# one can catenate several training strategies
training1 = ROOT.TString(
"LearningRate=1e-3,Momentum=0.9,"
"ConvergenceSteps=10,BatchSize=128,TestRepetitions=1,"
"MaxEpochs=20,WeightDecay=1e-4,Regularization=None,"
"Optimizer=ADAM,ADAM_beta1=0.9,ADAM_beta2=0.999,ADAM_eps=1.E-7," # ADAM default parameters
"DropConfig=0.0+0.0+0.0+0."
)
# training2 = ROOT.TString("LearningRate=1e-3,Momentum=0.9"
# "ConvergenceSteps=10,BatchSize=128,TestRepetitions=1,"
# "MaxEpochs=20,WeightDecay=1e-4,Regularization=None,"
# "Optimizer=SGD,DropConfig=0.0+0.0+0.0+0.")
# General Options.
dnnMethodName = ROOT.TString("DNN_CPU")
if useDLGPU:
arch = "GPU"
dnnMethodName = "DNN_GPU"
else:
arch = "CPU"
factory.BookMethod(
loader,
TMVA.Types.kDL,
dnnMethodName,
H=False,
V=True,
ErrorStrategy="CROSSENTROPY",
VarTransform="G",
WeightInitialization="XAVIER",
InputLayout="1|1|7",
BatchLayout="1|128|7",
Layout="DENSE|64|TANH,DENSE|64|TANH,DENSE|64|TANH,DENSE|64|TANH,DENSE|1|LINEAR",
TrainingStrategy=training1,
Architecture=arch,
)
# Keras DL
if useKeras:
ROOT.Info("TMVA_Higgs_Classification", "Building Deep Learning keras model")
# create Keras model with 4 layers of 64 units and relu activations
import tensorflow
from tensorflow.keras.models import Sequential
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.layers import Input, Dense
model = Sequential()
model.add(Dense(64, activation="relu", input_dim=7))
model.add(Dense(64, activation="relu"))
model.add(Dense(64, activation="relu"))
model.add(Dense(64, activation="relu"))
model.add(Dense(2, activation="sigmoid"))
model.compile(loss="binary_crossentropy", optimizer=Adam(learning_rate=0.001), weighted_metrics=["accuracy"])
model.save("model_higgs.h5")
model.summary()
if not os.path.exists("model_higgs.h5"):
raise FileNotFoundError("Error creating Keras model file - skip using Keras")
else:
# book PyKeras method only if Keras model could be created
ROOT.Info("TMVA_Higgs_Classification", "Booking Deep Learning keras model")
factory.BookMethod(
loader,
TMVA.Types.kPyKeras,
"PyKeras",
H=True,
V=False,
VarTransform=None,
FilenameModel="model_higgs.h5",
FilenameTrainedModel="trained_model_higgs.h5",
NumEpochs=20,
BatchSize=100,
)
# GpuOptions="allow_growth=True",
# ) # needed for RTX NVidia card and to avoid TF allocates all GPU memory
## Train Methods
# Here we train all the previously booked methods.
factory.TrainAllMethods()
## Test all methods
# Now we test and evaluate all methods using the test data set
factory.TestAllMethods()
factory.EvaluateAllMethods()
# after we get the ROC curve and we display
c1 = factory.GetROCCurve(loader)
c1.Draw()
# at the end we close the output file which contains the evaluation result of all methods and it can be used by TMVAGUI
# to display additional plots
outputFile.Close()
TRangeDynCast
ROOT::Detail::TRangeCast< T, true > TRangeDynCast
TRangeDynCast is an adapter class that allows the typed iteration through a TCollection.
Definition TCollection.h:358
TMVA::Factory
This is the main MVA steering class.
Definition Factory.h:80
Author
Harshal Shende

Definition in file TMVA_Higgs_Classification.py.