Cited By
View all- Cui CHu MWeir JWu T(2016)A recommendation system for meta-modelingExpert Systems with Applications: An International Journal10.1016/j.eswa.2015.10.02146:C(33-44)Online publication date: 15-Mar-2016
Accuracy vs. robustness: bi-criteria optimized ensemble of metamodels
Abstract
Simulation has been widely used in modeling engineering systems. A metamodel is a surrogate model used to approximate a computationally expensive simulation model. Extensive research has investigated the performance of different metamodeling techniques in terms of accuracy and/or robustness and concluded no model outperforms others across diverse problem structures. Motivated by this finding, this research proposes a bi-criteria (accuracy and robustness) optimized ensemble framework to optimally identify the contributions from each metamodel (Kriging, Support Vector Regression and Radial Basis Function), where uncertainties are modeled for evaluating robustness. Twenty-eight functions from the literature are tested. It is observed for most problems, a Pareto Frontier is obtained, while for some problems only a single point is obtained. Seven geometrical and statistical metrics are introduced to explore the relationships between the function properties and the ensemble models. It is concluded that the bi-criteria optimized ensembles render not only accurate but also robust metamodels.
References
[1]
Acar, E., M. Rais-Rohani. 2009. "Ensemble of Meta-models with Optimized Weight Factors." Structural and Multidisciplinary Optimization 37: 279--294.
[2]
Banks, J., J. S. Carson, B. L. Nelson, and D. M. Nicol. 2000. Discrete-Event System Simulation 3rd ed. Upper Saddle River, New Jersey: Prentice-Hall, Inc.
[3]
Bishop, C. M. 1995. Neural Networks for Pattern Recognition. New York: Oxford University Press.
[4]
Chang, C., C. Lin. 2011. "LIBSVM: A Library for Support Vector Machines." ACM Transactions on Intelligent Systems and Technology (TIST) 2: Article No. 27.
[5]
Clarke, S. M., J. H. Griebsch, T. W. Simpson. 2005. "Analysis of Support Vector Regression for Approximation of Complex Engineering Analyses." Journal of Mechanical Design 127(11):1077--1087.
[6]
Drucker, H., C. J. C. Burges, L. Kaufman, A. Smola, and V. Vapnik. 1996. "Support Vector Regression Machines," Advances in Neural Information Processing Systems 9, NIPS, 155--161, MIT Press.
[7]
Dyn, N., D. Levin, S. Rippa. 1986. "Numerical Procedures for Surface Fitting of Scattered Data by Radial Basis Functions." SIAM Journal on Scientific and Statistical Computing 7(2): 639--659.
[8]
Eglajs, V., P. Audze. 1977. "New Approach to the Design of Multifactor Experiments," Problems of Dynamics and Strengths 35: 104--107.
[9]
Fang, H., M. Rais-Rohani, Z. Liu, M. F. Horstemeyer. 2005. "A Comparative Study of Metamodeling Methods for Multiobjective Crashworthiness Optimization." Computers & Structures 83: 2121--2136.
[10]
Friedman, M. 1937. "The Use of Ranks to Avoid the Assumption of Normality Implicit in the Analysis of Variance," Journal of the American Statistical Association 32 (200): 675--701.
[11]
Hwang, C. L., A. S. M. Masud. 1979. "Multiple Objective Decision Making, Methods and Applications: A State-of-the-art Survey." Economics and Mathematical Systems 164.
[12]
Jin, R., X. Du, and W. Chen. 2003. "The Use of Metamodeling Techniques for Optimization under Uncertainty." Structural and Multidisciplinary Optimization 25: 99--116.
[13]
Kleijnen, J. 1995. "Theory and Methodology Verification and Validation of Simulation Models." European Journal of Operational Research 82:145--162.
[14]
Kohavi, R. 1995. "A Study of Cross-validation and Bootstrap for Accuracy Estimation and Model Selection." Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence 2 (12): 1137--1143.
[15]
Li, Y. F., S. H. Ng, M. Xie, and T. N. Goh. 2010. "A Systematic Comparison of Metamodeling Techniques for Simulation Optimization in Decision Support Systems." Applied Soft Computing 10: 1257--1273.
[16]
Liang, J. J., B. Y. Qu, and P. N. Suganthan. 2013. "Problem Definitions and Evaluation Criteria for the CEC 2013 Special Session on Real-Parameter Optimization." Technical Report Computational Intelligence Laboratory, Zhengzhou University, Zhengzhou China and Technical Report, Nanyang Technological University, Singapore.
[17]
Matheron, G. 1960. "Krigeage d'un Panneau Rectangulaire Par sa Périphérie," Note géostatistique no.28, CG, Ecole des Mines de Paris.
[18]
Nemenyi, P. B. 1963. "Distribution-free Multiple Comparisons," PhD thesis, Princeton University.
[19]
Nocedal, J., and S. J. Wright. 2006. Numerical Optimization. Springer.
[20]
Shan, S., and G. G. Wang. 2005. "An Efficient Pareto Set Identification Approach for Multiobjective Optimization on Black-Box Functions." Transactions of the ASME 127: 866--874.
[21]
Simpson, T. W., J. Peplinski, P. N. Koch, and J. K. Allen. 1997. "On the Use of Statistics in Design and the Implications for De terministic Computer Experiments." ASME Design Engineering Technical Conferences DTM-3881.
[22]
Simpson, T. W., J. Peplinski, P. N., Koch, and J. K. Allen. 2001. "Metamodels for Computer-based Engineering Design: Survey and Recommendations." Engineering with Computers 17(2): 129--150.
[23]
Vilalta, R., and Y. Drissi. 2002. "A Perspective View and Survey of Meta-Learning." Artificial Intelligence Review 18: 77--95.
[24]
Wang, G. G., and S. Shan. 2007. "Review of Metamodeling Techniques in Support of Engineering Design Optimization." Journal of Mechanical Design 370.
[25]
Zhang, S., P. Zhu, W. Chen, and P. Arendt. 2013. "Concurrent Treatment of Parametric Uncertainty and Metamodeling Uncertainty in Robust Design," Structural and Multidisciplinary Optimization 47: 63--76.
- Accuracy vs. robustness: bi-criteria optimized ensemble of metamodels
Recommendations
Ensembling Base Classifiers to Improve Predictive Accuracy
DCABES '15: Proceedings of the 2015 14th International Symposium on Distributed Computing and Applications for Business Engineering and Science (DCABES)The algorithm of ensembling base classifiers can improve predictive accuracy, and achieve a better generalization. However, the ensemble classification methods in literature have been used in more rule-based algorithms of classifier. This paper presents ...
A Comparative Study of Selected Classifiers with Classification Accuracy in User Profiling
CSIE '09: Proceedings of the 2009 WRI World Congress on Computer Science and Information Engineering - Volume 03In recent years the use of personalized service provisioning applications has been very popular. However, effective personalization cannot be achieved without accurate user profiles. In literature a number of classification algorithms have been used to ...
Improve Flow Accuracy and Byte Accuracy in Network Traffic Classification
ICIC '08: Proceedings of the 4th international conference on Intelligent Computing: Advanced Intelligent Computing Theories and Applications - with Aspects of Artificial IntelligenceMost of the current network traffic classification approaches employ single classifier method with achieving lower accuracy under small training set. Different from high flow accuracy, byte accuracy, as an important metric for network traffic ...
Comments
Information & Contributors
Information
Published In
Sponsors
Publisher
IEEE Press
Publication History
Published: 07 December 2014
Check for updates
Qualifiers
- Research-article
Conference
WSC '14
Sponsor:
Acceptance Rates
WSC '14 Paper Acceptance Rate 205 of 320 submissions, 64%;
Overall Acceptance Rate 3,413 of 5,075 submissions, 67%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 90Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 14 Sep 2024
Other Metrics
Citations
Cited By
View all- Cui CHu MWeir JWu T(2016)A recommendation system for meta-modelingExpert Systems with Applications: An International Journal10.1016/j.eswa.2015.10.02146:C(33-44)Online publication date: 15-Mar-2016
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in