Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.
Sign upGitHub is where the world builds software
Millions of developers and companies build, ship, and maintain their software on GitHub — the largest and most advanced development platform in the world.
Avoid creating redundant datasets #74
Open
Comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
If
enter_data()is called with the sametrain_pathtwice in a row and the data itself hasn't changed, a new Dataset does not need to be created.We should add a column which stores some kind of hash of the actual data. When a Dataset would be created, if the metadata and data hash are exactly the same as an existing Dataset, nothing should be added to the ModelHub database and the existing Dataset should be returned instead.