research-article

In-situ Stochastic Training of MTJ Crossbar based Neural Networks

Authors:

Ankur SrivastavaAuthors Info & Claims

ISLPED '18: Proceedings of the International Symposium on Low Power Electronics and Design

Article No.: 51, Pages 1 - 6

https://doi.org/10.1145/3218603.3218616

Published: 23 July 2018 Publication History

Abstract

Owing to high device density, scalability and non-volatility, Magnetic Tunnel Junction-based crossbars have garnered significant interest for implementing the weights of an artificial neural network. The existence of only two stable states in MTJs implies a high overhead of obtaining optimal binary weights in software. We illustrate that the inherent parallelism in the crossbar structure makes it highly appropriate for in-situ training, wherein the network is taught directly on the hardware. It leads to significantly smaller training overhead as the training time is independent of the size of the network, while also circumventing the effects of alternate current paths in the crossbar and accounting for manufacturing variations in the device. We show how the stochastic switching characteristics of MTJs can be leveraged to perform probabilistic weight updates using the gradient descent algorithm. We describe how the update operations can be performed on crossbars both with and without access transistors and perform simulations on them to demonstrate the effectiveness of our techniques. The results reveal that stochastically trained MTJ-crossbar NNs achieve a classification accuracy nearly same as that of real-valued-weight networks trained in software and exhibit immunity to device variations.

References

[1]

A. F. Vincent et al. 2015. Spin-transfer torque magnetic memory as a stochastic memristive synapse for neuromorphic systems. IEEE transactions on biomedical circuits and systems 9, 2 (2015), 166--174.

[2]

A. Sengupta et al. 2016. Hybrid Spintronic-CMOS Spiking Neural Network with On-Chip Learning: Devices, Circuits, and Systems. Physical Review Applied 6 (2016).

[3]

A. Sengupta et al. 2016. Probabilistic deep spiking neural systems enabled by magnetic tunnel junction. IEEE Transactions on Electron Devices 63 (2016), 2963--70.

[4]

D. C. Worledge et al. 2010. Switching distributions and write reliability of perpendicular spin torque MRAM. In Electron Devices Meeting (IEDM), 2010 IEEE International.

[5]

D. Querlioz et al. 2013. Immunity to device variations in a spiking neural network with memristive nanodevices. IEEE Transactions on Nanotechnology 12, 3 (2013).

Digital Library

[6]

D. Soudry et al. 2015. Memristor-based multilayer neural networks with online gradient descent training. IEEE transactions on neural networks and learning systems 26, 10 (2015), 2408--2421.

[7]

D. Zhang et al. 2016. All spin artificial neural networks based on compound spintronic synapse and neuron. IEEE transactions on biomedical circuits and systems 10, 4 (2016), 828--836.

[8]

D. Zhang et al. 2016. Stochastic spintronic device based synapses and spiking neurons for neuromorphic computation. In Nanoscale Architectures (NANOARCH), 2016 IEEE/ACM International Symposium on. IEEE, 173--178.

[9]

F. Li et al. 2016. Ternary weight networks. arXiv preprint arXiv:1605.04711 (2016).

[10]

H. Tomita et al. 2011. High-speed spin-transfer switching in GMR nano-pillars with perpendicular anisotropy. IEEE Transactions on Magnetics 47, 6 (2011), 1599--1602.

[11]

J. Dean et al. 2012. Large scale distributed deep networks. In Advances in Neural Information Processing Systems (NIPS). 1223--1231.

Digital Library

[12]

J. H. Lee et al. 2007. Defect-tolerant nanoelectronic pattern classifiers. International Journal of Circuit Theory and Applications 35, 3 (2007), 239--264.

Digital Library

[13]

J. Kim et al. 2015. A technology-agnostic MTJ SPICE model with user-defined dimensions for STT-MRAM scalability studies. In Custom Integrated Circuits Conference (CICC), 2015 IEEE. IEEE, 1--4.

[14]

J. Misra et al. 2010. Artificial neural networks in hardware: A survey of two decades of progress. 74, 1 (2010), 239--255.

Digital Library

[15]

K. L. Wang et al. 2013. Low-power non-volatile spintronic memory: STT-RAM and beyond. Journal of Physics D: Applied Physics 46, 7 (2013), 074003.

[16]

M. Courbariaux et al. 2015. Binaryconnect: Training deep neural networks with binary weights during propagations. In Advances in NIPS. 3123--3131.

Digital Library

[17]

M. Prezioso et al. 2015. Training and operation of an integrated neuromorphic network based on metal-oxide memristors. Nature 521, 7550 (2015), 61--64.

[18]

M. Rastegari et al. 2016. Xnor-net: Imagenet classification using binary convolutional neural networks. In European Conference on Computer Vision. Springer.

[19]

M. Suri et al. 2013. Bio-inspired stochastic computing using binary CBRAM synapses. IEEE Transactions on Electron Devices 60, 7 (2013), 2402--2409.

[20]

R. Hasan et al. 2014. Enabling back propagation training of memristor crossbar neuromorphic processors. In Neural Networks (IJCNN), 2014 International Joint Conference on. IEEE, 21--28.

[21]

R. Legenstein et al. 2005. What can a neuron learn with spike-timing-dependent plasticity? Neural computation 17, 11 (2005), 2337--2382.

Digital Library

[22]

S. Saïghi et al. 2015. Plasticity in memristive devices for spiking neural networks. 9 (2015).

[23]

W. Senn et al. 2005. Convergence of stochastic learning in perceptrons with binary synapses. Physical Review E 71, 6 (2005), 061907.

[24]

Y. LeCun et al. 1998. Gradient-based learning applied to document recognition. Proc. IEEE 86, 11 (1998), 2278--2324.

[25]

Y. Zhang et al. 2012. Asymmetry of MTJ switching and its implication to STT-RAM designs. In Proceedings of the Conference on Design, Automation and Test in Europe.

Digital Library

[26]

Z. Li et al. 2003. Magnetization dynamics with a spin-transfer torque. Physical Review B 68, 2 (2003), 024404.

[27]

M. Lichman. 2013. UCI Machine Learning Repository. (2013). http://archive.ics.uci.edu/ml

Cited By

Lone ALi HEl-Atab NSetti GFariborzi H(2023)Voltage-Gated Domain Wall Magnetic Tunnel Junction for Neuromorphic Computing ApplicationsIEEE Transactions on Electron Devices10.1109/TED.2023.332489870:12(6293-6300)Online publication date: Dec-2023
https://doi.org/10.1109/TED.2023.3324898
Lone ALi HEl-Atab NSetti GFariborzi H(2023)Voltage Gated Domain Wall Magnetic Tunnel Junction for Neuromorphic Computing Applications2023 IEEE 23rd International Conference on Nanotechnology (NANO)10.1109/NANO58406.2023.10231303(976-981)Online publication date: 2-Jul-2023
https://doi.org/10.1109/NANO58406.2023.10231303
Verma GBindal NNisar ADhull SKaushik B(2021)Advances in Neuromorphic Spin-Based Spiking Neural Networks: A reviewIEEE Nanotechnology Magazine10.1109/MNANO.2021.309821915:5(33-44)Online publication date: Oct-2021
https://doi.org/10.1109/MNANO.2021.3098219
Show More Cited By

Index Terms

In-situ Stochastic Training of MTJ Crossbar based Neural Networks
1. Computing methodologies
  1. Machine learning
    1. Learning paradigms
      1. Supervised learning
    2. Machine learning approaches
      1. Neural networks
2. Hardware
  1. Emerging technologies
    1. Analysis and design of emerging devices and systems
      1. Emerging architectures
    2. Spintronics and magnetic technologies
  2. Integrated circuits
    1. Semiconductor memory
      1. Non-volatile memory

Recommendations

In Situ Stochastic Training of MTJ Crossbars With Machine Learning Algorithms
Special Issue on HALO for Energy-Constrained On-Chip Machine Learning

Owing to high device density, scalability, and non-volatility, magnetic tunnel junction (MTJ)-based crossbars have garnered significant interest for implementing the weights of neural networks (NNs). The existence of only two stable states in MTJs ...
Energy-efficient Design of MTJ-based Neural Networks with Stochastic Computing

Hardware implementations of Artificial Neural Networks (ANNs) using conventional binary arithmetic units are computationally expensive, energy-intensive, and have large area overheads. Stochastic Computing (SC) is an emerging paradigm that replaces ...
Asymmetric dielectric breakdown behavior in MgO based magnetic tunnel junctions

The time-dependent dielectric breakdown phenomenon (TDDB) has been investigated in a series of nominally identical MgO based magnetic tunnel junctions (MTJs) by pulsed voltage endurance test. Results from the pulsed endurance test reveal that the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ISLPED '18: Proceedings of the International Symposium on Low Power Electronics and Design

July 2018

327 pages

ISBN:9781450357043

DOI:10.1145/3218603

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 July 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

ISLPED '18

Sponsor:

SIGDA

ISLPED '18: International Symposium on Low Power Electronics and Design

July 23 - 25, 2018

WA, Seattle, USA

Acceptance Rates

Overall Acceptance Rate 398 of 1,159 submissions, 34%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
252
Total Downloads

Downloads (Last 12 months)19
Downloads (Last 6 weeks)2

Reflects downloads up to 15 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Lone ALi HEl-Atab NSetti GFariborzi H(2023)Voltage-Gated Domain Wall Magnetic Tunnel Junction for Neuromorphic Computing ApplicationsIEEE Transactions on Electron Devices10.1109/TED.2023.332489870:12(6293-6300)Online publication date: Dec-2023
https://doi.org/10.1109/TED.2023.3324898
Lone ALi HEl-Atab NSetti GFariborzi H(2023)Voltage Gated Domain Wall Magnetic Tunnel Junction for Neuromorphic Computing Applications2023 IEEE 23rd International Conference on Nanotechnology (NANO)10.1109/NANO58406.2023.10231303(976-981)Online publication date: 2-Jul-2023
https://doi.org/10.1109/NANO58406.2023.10231303
Verma GBindal NNisar ADhull SKaushik B(2021)Advances in Neuromorphic Spin-Based Spiking Neural Networks: A reviewIEEE Nanotechnology Magazine10.1109/MNANO.2021.309821915:5(33-44)Online publication date: Oct-2021
https://doi.org/10.1109/MNANO.2021.3098219
Kingra SParmar VNegi SKhan SHudec BHou TSuri M(2020)Methodology for Realizing VMM with Binary RRAM Arrays: Experimental Demonstration of Binarized-ADALINE using OxRAM Crossbar2020 IEEE International Symposium on Circuits and Systems (ISCAS)10.1109/ISCAS45731.2020.9180915(1-5)Online publication date: Oct-2020
https://doi.org/10.1109/ISCAS45731.2020.9180915
Mondal ASrivastava A(2019)In Situ Stochastic Training of MTJ Crossbars With Machine Learning AlgorithmsACM Journal on Emerging Technologies in Computing Systems10.1145/330988015:2(1-29)Online publication date: 28-Mar-2019
https://dl.acm.org/doi/10.1145/3309880
Ostwal VZand RDeMara RAppenzeller J(2019)A Novel Compound Synapse Using Probabilistic Spin–Orbit-Torque Switching for MTJ-Based Deep Neural NetworksIEEE Journal on Exploratory Solid-State Computational Devices and Circuits10.1109/JXCDC.2019.29564685:2(182-187)Online publication date: Dec-2019
https://doi.org/10.1109/JXCDC.2019.2956468

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents