Repro Public Wall by https://padlet.com/mmueller26/xr5ryyosugpf Made with a warm hug en-us 2019-02-18 13:24:26 UTC 2026-01-09 14:28:07 UTC hello@padlet.com CoffeeScript mmueller26 https://padlet.com/mmueller26/xr5ryyosugpf/wish/2533258902 You can't reproduce an experiment in case you've spilled your beverage over your laptop.]]> 2023-03-27 15:01:05 UTC https://padlet.com/mmueller26/xr5ryyosugpf/wish/2533258902 Missing Code Repo https://padlet.com/mmueller26/xr5ryyosugpf/wish/2535001479 Use Github to have versioning and insight into the thought process.]]> 2023-03-28 13:05:15 UTC https://padlet.com/mmueller26/xr5ryyosugpf/wish/2535001479 Missing hyperparameters https://padlet.com/mmueller26/xr5ryyosugpf/wish/2535001755 2023-03-28 13:05:27 UTC https://padlet.com/mmueller26/xr5ryyosugpf/wish/2535001755 dataset not publicly available https://padlet.com/mmueller26/xr5ryyosugpf/wish/2535002295 2023-03-28 13:05:48 UTC https://padlet.com/mmueller26/xr5ryyosugpf/wish/2535002295 Missing software package information (e.g. version) https://padlet.com/mmueller26/xr5ryyosugpf/wish/2535002516 Even the of sequence in which software was installed can produce differences. Solution => Docker]]> 2023-03-28 13:05:59 UTC https://padlet.com/mmueller26/xr5ryyosugpf/wish/2535002516 Lack of records https://padlet.com/mmueller26/xr5ryyosugpf/wish/2535002657 During experimentation, parameters such as hyperparameter values and batch sizes change. Without properly logging the changes in these parameters, it becomes difficult to understand and replicate the model.]]> 2023-03-28 13:06:04 UTC https://padlet.com/mmueller26/xr5ryyosugpf/wish/2535002657 random seed setting https://padlet.com/mmueller26/xr5ryyosugpf/wish/2535002812 2023-03-28 13:06:11 UTC https://padlet.com/mmueller26/xr5ryyosugpf/wish/2535002812 Library versions https://padlet.com/mmueller26/xr5ryyosugpf/wish/2535002940 Different versions across libraries => Use dockers!

]]> 2023-03-28 13:06:16 UTC https://padlet.com/mmueller26/xr5ryyosugpf/wish/2535002940 GPU parallelism randomness https://padlet.com/mmueller26/xr5ryyosugpf/wish/2535003663 2023-03-28 13:06:44 UTC https://padlet.com/mmueller26/xr5ryyosugpf/wish/2535003663 Dataset https://padlet.com/mmueller26/xr5ryyosugpf/wish/2535004156 Dataset gets updated ]]> 2023-03-28 13:07:02 UTC https://padlet.com/mmueller26/xr5ryyosugpf/wish/2535004156 Competition and Safety https://padlet.com/mmueller26/xr5ryyosugpf/wish/2535004884 OpenAI purposefully did not reveal details about their newest GPT-4 model, citing competition and safety reasons]]> 2023-03-28 13:07:32 UTC https://padlet.com/mmueller26/xr5ryyosugpf/wish/2535004884 Libraries https://padlet.com/mmueller26/xr5ryyosugpf/wish/2535004914 Using different versions of libraries]]> 2023-03-28 13:07:33 UTC https://padlet.com/mmueller26/xr5ryyosugpf/wish/2535004914 Different implementation https://padlet.com/mmueller26/xr5ryyosugpf/wish/2535005045 2023-03-28 13:07:38 UTC https://padlet.com/mmueller26/xr5ryyosugpf/wish/2535005045 Publication bias https://padlet.com/mmueller26/xr5ryyosugpf/wish/2535005238 It is much easier to get published with new results compared to results that confirm the status-quo. Hence, there is an implicit incentive to "manufacture" surprising results and often reproduction studies can't achieve the same results

]]> 2023-03-28 13:07:45 UTC https://padlet.com/mmueller26/xr5ryyosugpf/wish/2535005238 Data Quality https://padlet.com/mmueller26/xr5ryyosugpf/wish/2535005434 Only raw data is available and preprocessing is not documented.]]> 2023-03-28 13:07:54 UTC https://padlet.com/mmueller26/xr5ryyosugpf/wish/2535005434 Non-Disclosure Agreements https://padlet.com/mmueller26/xr5ryyosugpf/wish/2535007025 For example in Database research (where a lot of machine learning experimentation happens) you are not allowed to release benchmarks of proprietary database engines. Access to technologies may only be granted with NDAs]]> 2023-03-28 13:09:00 UTC https://padlet.com/mmueller26/xr5ryyosugpf/wish/2535007025 Aspects affecting reproducibility https://padlet.com/mmueller26/xr5ryyosugpf/wish/2535008092 -Hardware equipments
-Resources (Money, Time etc.)
-Availability of datasets
-Environment and hyper-parameters
-Lack of implementation details
]]> 2023-03-28 13:09:31 UTC https://padlet.com/mmueller26/xr5ryyosugpf/wish/2535008092 Hardware resources https://padlet.com/mmueller26/xr5ryyosugpf/wish/2535009723 It's essentially impossible for anybody to replicate GPT3/GPT4 models since the computational power needed is out-of-reach]]> 2023-03-28 13:10:29 UTC https://padlet.com/mmueller26/xr5ryyosugpf/wish/2535009723 Dataset versioning https://padlet.com/mmueller26/xr5ryyosugpf/wish/2535009988 If data is preprocessed it is essential to keep a record of how the data was preprocessed especially when the preprocessing is non-deterministic.]]> 2023-03-28 13:10:39 UTC https://padlet.com/mmueller26/xr5ryyosugpf/wish/2535009988 Hardware specification https://padlet.com/mmueller26/xr5ryyosugpf/wish/2535011373 Including drivers etc.]]> 2023-03-28 13:11:32 UTC https://padlet.com/mmueller26/xr5ryyosugpf/wish/2535011373 Non deterministic algorithms https://padlet.com/mmueller26/xr5ryyosugpf/wish/2535011981 Use of algorithms where the output can change with the same input. Example - stochastic gradient descent]]> 2023-03-28 13:11:53 UTC https://padlet.com/mmueller26/xr5ryyosugpf/wish/2535011981