{ "cells": [ { "cell_type": "markdown", "metadata": { "pycharm": {} }, "source": [ "# Band Excitation data procesing\n", "### Suhas Somnath, Chris R. Smith, Stephen Jesse\n", "The Center for Nanophase Materials Science and The Institute for Functional Imaging for Materials
\n", "Oak Ridge National Laboratory
\n", "9/2/2020\n", "\n", "### Reference:\n", "This Jupyter notebook uses [pycroscopy](https://pycroscopy.github.io/pycroscopy/about.html) to analyze Band Excitation data. We request you to reference the [Arxiv paper](https://arxiv.org/abs/1903.09515) titled \"*USID and Pycroscopy - Open frameworks for storing and analyzing spectroscopic and imaging data*\" in your publications. \n", "\n", "#### Jupyter Notebooks:\n", "This is a Jupyter Notebook - it contains text and executable code `cells`. To learn more about how to use it, please see [this video](https://www.youtube.com/watch?v=jZ952vChhuI). Please see the image below for some basic tips on using this notebook.\n", "\n", "If you have any questions or need help running this notebook, please get in touch with your host if you are a users at the Center for Nanophase Materials Science (CNMS) or our [google group](https://groups.google.com/forum/#!forum/pycroscopy).\n", "\n", "![notebook_rules.png](https://raw.githubusercontent.com/pycroscopy/pycroscopy/master/jupyter_notebooks/notebook_rules.png)\n", "\n", "Image courtesy of Jean Bilheux from the [neutron imaging](https://github.com/neutronimaging/python_notebooks) GitHub repository." ] }, { "cell_type": "markdown", "metadata": { "pycharm": {} }, "source": [ "## Configure the notebook" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "pycharm": {} }, "outputs": [], "source": [ "# Make sure needed packages are installed and up-to-date\n", "import sys\n", "!conda install --yes --prefix {sys.prefix} numpy scipy matplotlib scikit-learn Ipython ipywidgets h5py \n", "!{sys.executable} -m pip install -U --no-deps BGlib # this will automatically install sidpy and pyUSID as well" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "pycharm": {} }, "outputs": [], "source": [ "# Ensure python 3 compatibility\n", "from __future__ import division, print_function, absolute_import\n", "\n", "# Import necessary libraries:\n", "# General utilities:\n", "import os\n", "\n", "# Computation:\n", "import numpy as np\n", "import h5py\n", "\n", "# Visualization:\n", "import matplotlib.pyplot as plt\n", "from IPython.display import display, HTML\n", "\n", "# The engineering components supporting BGlib:\n", "import sidpy\n", "import pyUSID as usid\n", "# Finally, BGlib itself\n", "from BGlib import be as belib\n", "\n", "# Make Notebook take up most of page width\n", "display(HTML(data=\"\"\"\n", "\n", "\"\"\"))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "pycharm": {} }, "outputs": [], "source": [ "# set up notebook to show plots within the notebook\n", "%matplotlib notebook" ] }, { "cell_type": "markdown", "metadata": { "pycharm": {} }, "source": [ "## Set some basic preferences\n", "This notebook performs some functional fitting whose duration can be substantially decreased by using more memory and CPU cores. We have provided default values below but you may choose to change them if necessary. Setting `max_cores` to `None` will allow usage of all but one CPU core for the computations. \n", "\n", "By default, results of the functional fitting will be written back to the same HDF5 file. However, if you prefer to write results into different HDF5 files, please set the `results_to_new_file` parameter to `True` instead. Users of the [DataFed](https://datafed.ornl.gov) Scientific Data Management System may want to set this parameter to `True`." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "pycharm": {} }, "outputs": [], "source": [ "max_mem = 1024*8 # Maximum memory to use, in Mbs. Default = 1024\n", "max_cores = None # Number of logical cores to use in fitting. None uses all but 2 available cores.\n", "results_to_new_file = False" ] }, { "cell_type": "markdown", "metadata": { "pycharm": {} }, "source": [ "## Make the data USID compatible\n", "Converting the raw data into a USID formatted hierarchical data format (HDF or .h5) file gives you access to the fast fitting algorithms and powerful analysis functions within the broader pycroscopy ecosystem\n", "\n", "#### H5 files:\n", "* are like smart containers that can store matrices with data, folders to organize these datasets, images, metadata like experimental parameters, links or shortcuts to datasets, etc.\n", "* are readily compatible with high-performance computing facilities\n", "* scale very efficiently from few kilobytes to several terabytes\n", "* can be read and modified using any language including Python, Matlab, C/C++, Java, Fortran, Igor Pro, etc.\n", "\n", "#### You can load either of the following:\n", "* Any .mat or .txt parameter file from the original experiment\n", "* A .h5 file generated from the raw data using BGlib - skips translation\n", "\n", "You can select desired file type by choosing the second option in the pull down menu on the bottom right of the file window" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "input_file_path = '/Users/syz/Desktop/BEPS_NDF/Example_1/20170225_MoTe2_S3_F4_vdc_sweep_contact_0001/20170225_MoTe2_S3_F4_vdc_sweep_contact_parms_0001.txt'" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "pycharm": {}, "scrolled": false }, "outputs": [], "source": [ "(data_dir, filename) = os.path.split(input_file_path)\n", "\n", "if input_file_path.endswith('.h5'):\n", " # No translation here\n", " h5_path = input_file_path\n", " force = True # Set this to true to force patching of the datafile.\n", " tl = belib.translators.LabViewH5Patcher()\n", " tl.translate(h5_path, force_patch=force)\n", "else:\n", " # Set the data to be translated\n", " data_path = input_file_path\n", "\n", " (junk, base_name) = os.path.split(data_dir)\n", "\n", " # Check if the data is in the new or old format. Initialize the correct translator for the format.\n", " if base_name == 'newdataformat':\n", " (junk, base_name) = os.path.split(junk)\n", " translator = belib.translators.BEPSndfTranslator(max_mem_mb=max_mem)\n", " else:\n", " translator = belib.translators.BEodfTranslator(max_mem_mb=max_mem)\n", " if base_name.endswith('_d'):\n", " base_name = base_name[:-2]\n", " # Translate the data\n", " print(translator)\n", " h5_path = translator.translate(data_path, show_plots=True, save_plots=False)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "folder_path, h5_raw_file_name = os.path.split(h5_path)\n", "h5_file = h5py.File(h5_path, 'r+')\n", "print('Working on:\\n' + h5_path)\n", "\n", "h5_main = usid.hdf_utils.find_dataset(h5_file, 'Raw_Data')[0]" ] }, { "cell_type": "markdown", "metadata": { "pycharm": {} }, "source": [ "##### Inspect the contents of this h5 data file\n", "The file contents are stored in a tree structure, just like files on a conventional computer.\n", "The data is stored as a 2D matrix (position, spectroscopic value) regardless of the dimensionality of the data. Thus, the positions will be arranged as row0-col0, row0-col1.... row0-colN, row1-col0.... and the data for each position is stored as it was chronologically collected \n", "\n", "The main dataset is always accompanied by four ancillary datasets that explain the position and spectroscopic value of any given element in the dataset." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "pycharm": {}, "scrolled": false }, "outputs": [], "source": [ "print('Datasets and datagroups within the file:\\n------------------------------------')\n", "sidpy.hdf.hdf_utils.print_tree(h5_file)\n", " \n", "print('\\nThe main dataset:\\n------------------------------------')\n", "print(h5_main)\n", "\n", "print('\\nMetadata or attributes in the measurement datagroup\\n------------------------------------')\n", "for key, val in sidpy.hdf.hdf_utils.get_attributes(h5_main.parent.parent).items():\n", " print('{} : {}'.format(key, val))" ] }, { "cell_type": "markdown", "metadata": { "pycharm": {} }, "source": [ "## Get some basic parameters from the H5 file\n", "This information will be vital for futher analysis and visualization of the data" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "pycharm": {} }, "outputs": [], "source": [ "h5_pos_inds = h5_main.h5_pos_inds\n", "pos_dims = h5_main.pos_dim_sizes\n", "pos_labels = h5_main.pos_dim_labels\n", "print(pos_labels, pos_dims)\n", "\n", "h5_meas_grp = h5_main.parent.parent\n", "\n", "parm_dict = sidpy.hdf.hdf_utils.get_attributes(h5_meas_grp)\n", "\n", "expt_type = sidpy.hdf.hdf_utils.get_attr(h5_file, 'data_type')\n", "\n", "is_ckpfm = expt_type == 'cKPFMData'\n", "if is_ckpfm:\n", " num_write_steps = parm_dict['VS_num_DC_write_steps']\n", " num_read_steps = parm_dict['VS_num_read_steps']\n", " num_fields = 2\n", " \n", "if expt_type != 'BELineData':\n", " vs_mode = sidpy.hdf.hdf_utils.get_attr(h5_meas_grp, 'VS_mode')\n", " try:\n", " field_mode = sidpy.hdf.hdf_utils.get_attr(h5_meas_grp, 'VS_measure_in_field_loops')\n", " except KeyError:\n", " print('field mode could not be found. Setting to default value')\n", " field_mode = 'out-of-field'\n", " try:\n", " vs_cycle_frac = sidpy.hdf.hdf_utils.get_attr(h5_meas_grp, 'VS_cycle_fraction')\n", " except KeyError:\n", " print('VS cycle fraction could not be found. Setting to default value')\n", " vs_cycle_frac = 'full'" ] }, { "cell_type": "markdown", "metadata": { "pycharm": {} }, "source": [ "## Visualize the raw data\n", "Use the sliders below to visualize spatial maps (2D only for now), and spectrograms.\n", "For simplicity, all the spectroscopic dimensions such as frequency, excitation bias, cycle, field, etc. have been collapsed to a single slider." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "pycharm": {}, "scrolled": true }, "outputs": [], "source": [ "fig = belib.viz.be_viz_utils.jupyter_visualize_be_spectrograms(h5_main)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Fit the Band Excitation (BE) spectra\n", "Fit each of the acquired spectra to a simple harmonic oscillator (SHO) model to extract the following information regarding the response:\n", "* Oscillation amplitude\n", "* Phase\n", "* Resonance frequency\n", "* Quality factor\n", "\n", "By default, the cell below will take any previous result instead of re-computing the SHO fit" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "pycharm": {} }, "outputs": [], "source": [ "sho_fit_points = 5 # The number of data points at each step to use when fitting\n", "sho_override = False # Force recompute if True\n", "\n", "h5_sho_targ_grp = None\n", "if results_to_new_file:\n", " h5_sho_file_path = os.path.join(folder_path, \n", " h5_raw_file_name.replace('.h5', '_sho_fit.h5'))\n", " print('\\n\\nSHO Fits will be written to:\\n' + h5_sho_file_path + '\\n\\n')\n", " f_open_mode = 'w'\n", " if os.path.exists(h5_sho_file_path):\n", " f_open_mode = 'r+'\n", " h5_sho_file = h5py.File(h5_sho_file_path, mode=f_open_mode)\n", " h5_sho_targ_grp = h5_sho_file\n", " \n", "sho_fitter = belib.analysis.BESHOfitter(h5_main, cores=max_cores, verbose=False, h5_target_group=h5_sho_targ_grp)\n", "sho_fitter.set_up_guess(guess_func=belib.analysis.be_sho_fitter.SHOGuessFunc.complex_gaussian,\n", " num_points=sho_fit_points)\n", "h5_sho_guess = sho_fitter.do_guess(override=sho_override)\n", "sho_fitter.set_up_fit()\n", "h5_sho_fit = sho_fitter.do_fit(override=sho_override)\n", "h5_sho_grp = h5_sho_fit.parent" ] }, { "cell_type": "markdown", "metadata": { "pycharm": {} }, "source": [ "## Visualize the SHO results\n", "Here, we visualize the parameters for the SHO fits. BE-line (3D) data is visualized via simple spatial maps of the SHO parameters while more complex BEPS datasets (4+ dimensions) can be visualized using a simple interactive visualizer below. \n", "\n", "You can choose to visualize the guesses for SHO function or the final fit values from the first line of the cell below.\n", "\n", "Use the sliders below to inspect the BE response at any given location. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "pycharm": {} }, "outputs": [], "source": [ "h5_sho_spec_inds = h5_sho_fit.h5_spec_inds\n", "sho_spec_labels = h5_sho_fit.spec_dim_labels\n", "\n", "if is_ckpfm:\n", " # It turns out that the read voltage index starts from 1 instead of 0\n", " # Also the VDC indices are NOT repeating. They are just rising monotonically\n", " write_volt_index = np.argwhere(sho_spec_labels == 'write_bias')[0][0]\n", " read_volt_index = np.argwhere(sho_spec_labels == 'read_bias')[0][0]\n", " h5_sho_spec_inds[read_volt_index, :] -= 1\n", " h5_sho_spec_inds[write_volt_index, :] = np.tile(np.repeat(np.arange(num_write_steps), num_fields), num_read_steps)\n", "\n", "(Nd_mat, success, nd_labels) = usid.hdf_utils.reshape_to_n_dims(h5_sho_fit, get_labels=True)\n", "print('Reshape Success: ' + str(success))\n", "\n", "print(nd_labels)\n", "print(Nd_mat.shape)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "pycharm": {}, "scrolled": true }, "outputs": [], "source": [ "use_sho_guess = False\n", "use_static_viz_func = True\n", "\n", "if use_sho_guess:\n", " sho_dset = h5_sho_guess\n", "else:\n", " sho_dset = h5_sho_fit\n", " \n", "if expt_type == 'BELineData' or len(pos_dims) != 2:\n", " use_static_viz_func = True\n", " step_chan = None\n", "else:\n", " if vs_mode not in ['AC modulation mode with time reversal', \n", " 'DC modulation mode']:\n", " use_static_viz_func = True\n", " else:\n", " if vs_mode == 'DC modulation mode':\n", " step_chan = 'DC_Offset'\n", " else:\n", " step_chan = 'AC_Amplitude'\n", "if not use_static_viz_func:\n", " try:\n", " # use interactive visualization\n", " belib.viz.be_viz_utils.jupyter_visualize_beps_sho(sho_dset, step_chan)\n", " except:\n", " raise\n", " print('There was a problem with the interactive visualizer')\n", " use_static_viz_func = True\n", "else:\n", " chan_grp = h5_main.parent\n", " meas_grp = chan_grp.parent\n", " # show plots of SHO results vs. applied bias\n", " figs = belib.viz.be_viz_utils.visualize_sho_results(sho_dset, show_plots=True, save_plots=False, \n", " expt_type=expt_type, meas_type=vs_mode, \n", " field_mode=field_mode)" ] }, { "cell_type": "markdown", "metadata": { "pycharm": {} }, "source": [ "## Fit loops to a function\n", "This is applicable only to DC voltage spectroscopy datasets from BEPS. The PFM hysteresis loops in this dataset will be projected to maximize the loop area and then fitted to a function.\n", "\n", "Note: This computation generally takes a while for reasonably sized datasets." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "pycharm": {}, "scrolled": false }, "outputs": [], "source": [ "# Do the Loop Fitting on the SHO Fit dataset\n", "loop_success = False\n", "\n", "h5_loop_group = None\n", "if results_to_new_file:\n", " h5_loop_file_path = os.path.join(folder_path, \n", " h5_raw_file_name.replace('.h5', '_loop_fit.h5'))\n", " print('\\n\\nLoop Fits will be written to:\\n' + h5_loop_file_path + '\\n\\n')\n", " f_open_mode = 'w'\n", " if os.path.exists(h5_loop_file_path):\n", " f_open_mode = 'r+'\n", " h5_loop_file = h5py.File(h5_loop_file_path, mode=f_open_mode)\n", " h5_loop_group = h5_loop_file\n", " \n", "loop_fitter = belib.analysis.BELoopFitter(h5_sho_fit, expt_type, vs_mode, vs_cycle_frac,\n", " cores=max_cores, h5_target_group=h5_loop_group, \n", " verbose=False)\n", "loop_fitter.set_up_guess()\n", "h5_loop_guess = loop_fitter.do_guess(override=False)\n", "# Calling explicitely here since Fitter won't do it automatically\n", "h5_guess_loop_parms = loop_fitter.extract_loop_parameters(h5_loop_guess)\n", "\n", "loop_fitter.set_up_fit()\n", "h5_loop_fit = loop_fitter.do_fit(override=False)\n", "h5_loop_group = h5_loop_fit.parent\n", "loop_success = True" ] }, { "cell_type": "markdown", "metadata": { "pycharm": {} }, "source": [ "## Prepare datasets for visualization" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "pycharm": {} }, "outputs": [], "source": [ "# Prepare some variables for plotting loops fits and guesses\n", "# Plot the Loop Guess and Fit Results\n", "if loop_success:\n", " h5_projected_loops = usid.USIDataset(h5_loop_guess.parent['Projected_Loops'])\n", " h5_proj_spec_inds = h5_projected_loops.h5_spec_inds\n", " h5_proj_spec_vals = h5_projected_loops.h5_spec_vals\n", "\n", " # reshape the vdc_vec into DC_step by Loop\n", " sort_order = usid.hdf_utils.get_sort_order(h5_proj_spec_inds)\n", " dims = usid.hdf_utils.get_dimensionality(h5_proj_spec_inds[()], \n", " sort_order[::-1])\n", " vdc_vec = np.reshape(h5_proj_spec_vals[h5_proj_spec_vals.attrs['DC_Offset']], dims).T\n", "\n", " #Also reshape the projected loops to Positions-DC_Step-Loop\n", " # Also reshape the projected loops to Positions-DC_Step-Loop\n", " proj_nd = h5_projected_loops.get_n_dim_form()\n", " proj_3d = np.reshape(proj_nd, [h5_projected_loops.shape[0], \n", " proj_nd.shape[2], -1])" ] }, { "cell_type": "markdown", "metadata": { "pycharm": {} }, "source": [ "## Visualize Loop fits" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "pycharm": {}, "scrolled": false }, "outputs": [], "source": [ "use_static_plots = True\n", "if loop_success:\n", " if not use_static_plots:\n", " try:\n", " fig = belib.viz.be_viz_utils.jupyter_visualize_beps_loops(h5_projected_loops, h5_loop_guess, h5_loop_fit)\n", " except:\n", " print('There was a problem with the interactive visualizer')\n", " use_static_plots = True\n", " if use_static_plots:\n", " for iloop in range(h5_loop_guess.shape[1]):\n", " fig, ax = belib.viz.be_viz_utils.plot_loop_guess_fit(vdc_vec[:, iloop], proj_3d[:, :, iloop], \n", " h5_loop_guess[:, iloop], h5_loop_fit[:, iloop],\n", " title='Loop {} - All Positions'.format(iloop))" ] }, { "cell_type": "markdown", "metadata": { "pycharm": {} }, "source": [ "## Loop Parameters\n", "We will now load the loop parameters caluculated from the fit and plot them." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "pycharm": {}, "scrolled": false }, "outputs": [], "source": [ "h5_loop_parameters = h5_loop_group['Fit_Loop_Parameters']\n", "fig = belib.viz.be_viz_utils.jupyter_visualize_parameter_maps(h5_loop_parameters)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "pycharm": {}, "scrolled": false }, "outputs": [], "source": [ "map_parm = 'Work of Switching'\n", "plot_cycle = 0\n", "plot_position = (int(pos_dims[0]/2), int(pos_dims[1]/2))\n", "plot_bias_step = 0\n", "\n", "fig = belib.viz.be_viz_utils.plot_loop_sho_raw_comparison(h5_loop_parameters, h5_sho_grp, h5_main,\n", " selected_loop_parm=map_parm, \n", " selected_loop_cycle=plot_cycle, \n", " selected_loop_pos=plot_position, \n", " selected_step=plot_bias_step)\n" ] }, { "cell_type": "markdown", "metadata": { "pycharm": {} }, "source": [ "## Save and close\n", "* Save the .h5 file that we are working on by closing it.
\n", "* Also, consider exporting this notebook as a notebook or an html file.
To do this, go to File >> Download as >> HTML\n", "* Finally consider saving this notebook if necessary" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "pycharm": {} }, "outputs": [], "source": [ "h5_file.close()\n", "if results_to_new_file:\n", " h5_sho_fit.file.close()\n", " h5_loop_fit.file.close()" ] } ], "metadata": { "anaconda-cloud": {}, "kernelspec": { "display_name": "Python [default]", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.5.5" }, "widgets": { "state": { "626c09f4ed724d658d702180fe718a7f": { "views": [ { "cell_index": 12 } ] } }, "version": "1.2.0" } }, "nbformat": 4, "nbformat_minor": 1 }