A tool to find similar supercomputer applications using system monitoring time-series data and structural pattern recognition.
- Use R ecp package to divide sensor time-series into intervals. https://cran.r-project.org/web/packages/ecp/index.html
- Extract structural features from each subinterval. Idea from Generalized Feature Extraction for Structural Pattern Recognition in Time-Series Data Robert T. Olszewski February 2001 https://www.cs.cmu.edu/~bobski/pubs/tr01108-twosided.pdf
- Encode extracted structural primitives with letters.
- Use string distance metrics to tell distance between two supercomputer jobs.
- Train a classifier to discriminate between similar and not similar pairs of jobs.
src/pipeline - Workflow scripts:
load_raw_job_from_db.py- Load sensor data from database.extract_features.py- Preprocess data, find intervals with ecp, extract structural features.classify.py- Using encoded jobs and labeled data, calculate string distances, classify jobs and measure accuracy.
Also some useful additional tools:
get_job_info.py- Get job info from db for specified job_id.show_last_jobs.py- Show 10 last jobs of specified length.read_job_string_dict.py- Show extracted string features for specified job_id.
Scripts use a config file:
config/config.py- Reads the JSON config to python.config/hpc_similarity.conf- JSON config file.
src/feature_extractor - Feature extraction library:
preprocessing.py- All preprocessing done to sensor time series.structures.py- Classes for all structural primitives.feature_extraction.py- Extraction of optimal structural features.change_point_detection.py- Finding change points with ecp.plotting.py- Useful plotting functions.
data/labeled_node_average_data - Job data downloaded from db. For every job - one pickle file 'job{job_id}.pickle'
data/train_data - Contains train.json and test.json - JSON files with labeled pairs looking like this:
{'job1:job2': true, 'job2:job3':false, 'job1:job3': true}
These are train and test datasets respectively.
data/extracted - Feature extraction results for every job. For every job - one pickle file 'job{job_id}.pickle.
It contains fields 'preprocessed_job_data', 'change_points', 'change_points_t', 'superstructure_dict', 'encoded_structure_dict'.
data/string_extracted - Only extracted dict with extracted strings for every sensor. One pickle file for every job.