# data[].attributes.headline
Actionable Items R1 - Community Building

# data[].attributes.subject
Actionable Items R1 - Community Building

# data[].attributes.body
Philip Dart (UniMelb RSE Convenor)
pdart@unimelb.edu.au
Thank you to everyone who participated.
In brief, and in no particular order:
1) Piggy-back off existing events, such as training, to introduce community elements.
2) Hold broad-based events which are likely to achieve critical mass within the institution, e.g. get your (RSE) development questions answered.
3) Use focussed events, e.g. in Bioinformatics (or even particular Bioinformatics systems) focussed more broadly on the research community but which will bring in the relevant RSEs.
(Perhaps more to add when I get the zoom chat transcript. :-)

# data[].attributes.body
Thanks to Cynthia Huang @cynthiahqy

# data[].attributes.wish_content.attachment_props.text
Some short notes on the really great Research Software Engineering (RSE) Strategy & Policy Panel discussion just now on the final day of #RSEAA #unconference — thanks to @rse_aunz and @RSE_Asia for organising; (and @visnut for registering me!)

# data[].attributes.wish_content.attachment_props.original_text
Some short notes on the really great Research Software Engineering (RSE) Strategy & Policy Panel discussion just now on the final day of #RSEAA #unconference — thanks to @rse_aunz and @RSE_Asia for organising; (and @visnut for registering me!) https://t.co/mMfoEtbYqI

# data[].attributes.attachment_link.embed_code
Some short notes on the really great Research Software Engineering (RSE) Strategy & Policy Panel discussion just now on the final day of #RSEAA #unconference — thanks to @rse_aunz and @RSE_Asia for organising; (and @visnut for registering me!) pic.twitter.com/mMfoEtbYqI
— Cynthia Huang (@cynthiahqy) September 16, 2022

# data[].attributes.attachment_link.metadata.attachment_props_source.data.text
Some short notes on the really great Research Software Engineering (RSE) Strategy & Policy Panel discussion just now on the final day of #RSEAA #unconference — thanks to @rse_aunz and @RSE_Asia for organising; (and @visnut for registering me!) https://t.co/mMfoEtbYqI

# data[].attributes.attachment_link.metadata.attachment_props.original_text
Some short notes on the really great Research Software Engineering (RSE) Strategy & Policy Panel discussion just now on the final day of #RSEAA #unconference — thanks to @rse_aunz and @RSE_Asia for organising; (and @visnut for registering me!) https://t.co/mMfoEtbYqI

# data[].attributes.attachment_link.metadata.attachment_props.text
Some short notes on the really great Research Software Engineering (RSE) Strategy & Policy Panel discussion just now on the final day of #RSEAA #unconference — thanks to @rse_aunz and @RSE_Asia for organising; (and @visnut for registering me!)

# data[].attributes.headline
What do you think we should we doing next?

# data[].attributes.subject
What do you think we should we doing next?

# data[].attributes.body
- More RSE collaborations!
- supporting others more ally, those who have similar problems, colleagues across the spectrum
- The change is coming, maintain hope
- keep reaching out and engaging, communicate and create awareness RSEs

# data[].attributes.headline
Given a magic wand and the passage of time, how would you describe the ideal life of an RSE in 10 years time?

# data[].attributes.subject
Given a magic wand and the passage of time, how would you describe the ideal life of an RSE in 10 years time?

# data[].attributes.body
- You are much more valued in academia, including the pay checks.
- Teams of RSEs at research software organisations collaboration
- Build capacity for RSEs to get absorbed by any other teams. Transferrable skills.
- From academia to catch up with industry. RSEs will be easy to find, RSEs will like to stay in academia because they are well rewarded. The Definition is not a challenge anymore. A clear trajectory of development well defined. Recognise the work.
- The community to be collaborative and solve complex problems together.
- RSEs having positions with autonomy and decide where are their contributions more valuable.
- RSEs are not just supporting, they are leading

# data[].attributes.headline
how we can involve more influential RSE advocates to reach decision-makers?

# data[].attributes.subject
how we can involve more influential RSE advocates to reach decision-makers?

# data[].attributes.body
- Advocating, what RSEs enable, rather than the role. Going back to Software as infrastructure.
- So decision makers will see the need for training is relevant
- For the RSE community to be part of other influential communities, meet the right people.
-

# data[].attributes.headline
What skills should we be emphasising during early career development?

# data[].attributes.subject
What skills should we be emphasising during early career development?

# data[].attributes.body
- Statistics
- Programming
- Formal logic, computational thinking
- Research methodology, research hypothesis, research methods, Practical research experience
- version control
- testing (continuous integration)
-  Operating systems (path locations, memory, processors, etc)
- collaboration

# data[].attributes.headline
What is the importance of having RSE contributions evaluated for career progression?

# data[].attributes.subject
What is the importance of having RSE contributions evaluated for career progression?

# data[].attributes.body
- critical
- we are not the only community facing this challenge. We need to join forces
- RSEs are highly trained.
- Have annual performance reviews.
- The value needs to be more visible. We need to re evaluate what we value. Not just the output, but the process

# data[].attributes.body
- RSEs are more inclusive, enabling research
- There is definitely a lot of overlap between both
- Many definitions are conflicting, eg. Software engineer, Computer scientist, data science engineer, scientist-engineers
-

# data[].attributes.headline
What do we have to do more of to recognise the role of RSEs?

# data[].attributes.subject
What do we have to do more of to recognise the role of RSEs?

# data[].attributes.body
Non-traditional metrics:
- Valuing the software for it's impact
- Valuing the skills that make the software too
- One approach is that the software itself is the publication
- Software is the core part of the work of RSEs
- without software there is no other outputs
- Recognise the value at the learning stage, at university courses, create awareness of the importance of the role, at the undergrad level.
- encourage citation
- Funders look at the utility of the software, a community,  tracked development
- Have RSE flags in other conferences (specific domains, data science in general)

# data[].attributes.headline
What has improved in recognition of RSE the last few years ?

# data[].attributes.subject
What has improved in recognition of RSE the last few years ?

# data[].attributes.body
- Increased awareness of the role
- In Australia there is more funding for research infrastructure in the last few years. Software is infrastructure
- There are more jobs being advertised with the role title RSE
- The important of training and maintaining the group of people who have RSE skills. There is a lot of demand specially in academia. The skill of "solving problems " is emphasised in RSEs
- Use "software journals" more and let people know they exist

# data[].attributes.body
1) Platforms like Slack or Discord could be a great way to build community in a less formal way - many RSEs don't have many peers doing the same sort of work. More permanent platforms like https://discourse.pangeo.io can also be useful
2) RSEs often find themselves working with researchers from different disciplines - and these disciplines will have their own standards and language for talking about data and statistics. It's a challenge to respect the disciplines' own communities of practice while also building bridges between them
3) boundaries between RSE, data scientists, data engineers and software engineers can be blurred, which can cause problems for recognition

# data[].attributes.body
We very much spoke about the issues that undercut software as having the value expected of a first-class output.
- importance to career of recognising the work that goes into software - it's often invisible
- lifetime of sw - can outlive the group that originated it.
- Journal of Open Source Software (JOSS) allows pairing of sw/repo with a paper - which gives a lot of people somewhere to begin when assessing he value of software.
- important to use DOIs, so on GitHub can use zenodo.
- maintenance of sw - by it's nature , sw is mutable, so releases have to made carefully to snapshot a codebase  properly
- code availability should be a norm, along with data availability
- some fields are at the forefront of valuing software - eg: BioInformatics, Statisitics/Mathematics
- fore reviweres, sw can be problematic, especially if not reproducible environment.
- Monash has gone some way down this road: https://robjhyndman.com/files/EBS_research_software.pdf
Interesting question raised - does software necessarily offer an original contribution to human knowledge? how to assess this? This is often the core principle for other research outputs, after all.
- question - do the first entrants in this area have to overachieve, to really prove the point?

some other links, without exact context:
- https://en.wikipedia.org/wiki/ACM_Transactions_on_Mathematical_Software
- https://zenodo.org/

# data[].attributes.body
1. Why we need the process, what are we supporting continuation of, why it is essential to maintain - sometimes the outcome of the software is more important than the processes - e.g. it may be easier to convince to fund if ‘it enhances our reputation’ vs ‘we want this software to continue to exist’.

2. Proactive inclusion of goals for software in grants / project objectives - catch early - regular review - e.g. is it just a contractual obligation? Is it intended to continue after the project? This can change as a project continues (influenced by usage / metrics / research impact). Software management plan in funding submissions.
The team who enables the run of services, is not always the team that considers all the grant application process.

3. Making clear the many types of maintenance and many reasons for the maintenance to those that depend on the software - hardware, dbs, services, OSS, features, security / dep updates. Not always the most productive to group them.

Types of maintenance collected from the session:

Hardware

Sysadmin

own/internal software

Databases

hosted (self or otherwise) services

open source software

Community management

Code maintenance

Docs maintenance

Dataset curation and data management

Dependency maintenance

Learn to delegate other tasks, and commit to your maintenance tasks

Reviews - keeping tabs on whether it’s being used, whether it still has value

Some possible justifications for maintenance:

Maintain security - insecure software is dangerous and likely won’t be used.

Ensure sensitive data is protected - ethical and legal implications.

Maintaining software used by internal and external people - reputation of the supporting institution. +1

General engagement / ongoing research - not really fair to put costs on one researcher or project.

Graceful shutdown of project - simply dropping the software support makes it very hard to pick up again or know the state of the software.

Prepare to merge into or for sponsor by another group or org - valuable software could be transferred to another group or org to continue support.

Prepare for commercialisation - better chance to move to commercialisation if software is in a reasonable state.

Normal part of software dev - convince fund controllers to do ‘good’ software dev practices.

Operations and scaling software services - up or down usage, hard to know what’s being used if there’s no on-going monitoring.

# data[].attributes.body
Should promote collaboration and sharing across the RSE community. To achieve these objectives we feel that the top priority features are as follows:

1. User profiles

2. Forums

3. Job ads

4. Event registry

5. News and Links

6. Project profiles

7. Subscribing to updates

8. Publicly accessible pages

Biggest question - How do we get this project started?

# data[].attributes.headline
#TakeHome on  #BridgingTheGap between #Biologists and #Programmers 

# data[].attributes.subject
#TakeHome on  #BridgingTheGap between #Biologists and #Programmers 

# data[].attributes.body
Just finished organising an #unconference #session on #BridgingTheGap between #Biologists and #Programmers #RSEAA2022 #events #Australia We have had wonderful inputs. Thank you #RSEAA Rowland Mosbergen Paula Andrea Martinez

The key discussions include:

#Collaborations and #Communication which are the key and #COVID19 has been a blessing in disguise for biologists to come closer to programming.

Terry opines that key challenges should be sustainable and there must be open source practices all through!

Estelle and Harry highlight the need for udnerstanding each others' goals

Thom gives wonderful experiences from his company and how problem solving helps overcome the fear of taking up coding

Christopher feels relying in the expertise
Annajiat Alim, eslam and K Nishodo chip in and echo the sentiments

# data[].attributes.headline
S2R2 / S2R4 - performance review and career progression

# data[].attributes.subject
S2R2 / S2R4 - performance review and career progression

# data[].attributes.body
1) Performance reviews can be a way to plan for training, upskilling, professional development, opportunities to collaborate with different groups of researchers, conferences
2) Academic/researcher performance checklists can be adapted for RSE activities - ie code review can be seen as mentoring - which provides a framework for recognition
3) Frequent check-ins with line managers are really important

# data[].attributes.headline
S2R3: What would a secure RSE postion look like?

# data[].attributes.subject
S2R3: What would a secure RSE postion look like?

# data[].attributes.body
(full notes in attached file)

- At what level are services provided and what's the vision? To centralise or not to centralise? Centralisation requires a very clear goal beyond "provide IT/project support".
- Different institutions are trying different things, largely in isolation, but it's good that lots of experimentation is going on
- academic vs professional - different versions of secure and performance expectations. There is no Clear Path career-wise as an RSE
- central vs embedded - tied to grants / projects so rolling contracts - good and bad - can be good to be embedded but can be siloed, sometimes to be more secure requires being centralised, but that can lead to detachment from projects
- What value can we have by getting together in our own institutions? Making local chapters? Increasing visibility and value!

# data[].attributes.attachment_link.readable_text
## S2R3 What would a secure RSE position look like?
- lots of people are very interested in what this would look like
- interrelates with question of what a secure academic position would look like!
- so many unsecure positions! We've all been in them - the RA and contractor positions where you can just get lost or trapped
- we've got to treat RSEs well to keep them - it's hard because the roles aren't really recognised by funding bodies
- University of Melbourne
	- Philip Dart is RSE convenor at UniMelb, did a report about RSEs 2019-2020
	- UniMelb has at least 150 RSEs but they don't track them and really have no idea how many there are  - so many different kinds of positions, so many job titles
	- As there is no evident problem going on it's hard to justify efforts to improve matters
	- There's a precedent in the data specialist space, trying to formalise a similar specialist system that is not a centralised model, probably getting faculties to underwrite positions for longer term contracts and in the future continuing positions
- University of Sydney has recently centralised RSE in a five year program with five year remote contracts - they got to this position because things were so bad that all faculties agreed to a drastic shift
- Princeton employs RSEs centrally and rents them out to people at half cost
- Sometimes you can cobble together pockets of money into a full time position but it's hard work and the pieces often don't line up
- In some places it's expected for RSEs to be involved in some of the publications, in some it's not - getting agreement on norms like this is hard
- Centralisation of services can have a bad reputation in some places at some times
- Groups within universities and also outside of universities that do project based consulting with research proejcts
- Isolation of RSEs is a problem - the worst number of devs is wrong
- Can't really cause change without control of the money
- So many people on rolling contracts, renewed year after year, the only reason to not make them continuing is because they can get away with it
- The silos and communication problem
- need to recognise that the service is even there in the first place in order for it to be centralise - this could be helped by the model of a central(ish) group being seconded and embedded in a variety of places
- grants don't list RSE services as costs in the same way that they'd list physical equipment / reactants as costs! (Gen's point)


Takeaways:
- At what level are services provided and what's the vision? To centralise or not to centralise? Centralisation requires a very clear goal beyond "provide IT/project support".
- Different institutions are trying different things, largely in isolation, but it's good that lots of experimentation is going on
- academic vs professional - different versions of secure and performance expectations. There is no Clear Path career-wise as an RSE
- central vs embedded - tied to grants / projects so rolling contracts - good and bad - can be good to be embedded but can be siloed, sometimes to be more secure requires being centralised, but that can lead to detachment from projects
- What value can we have by getting together in our own institutions? Making local chapters? Increasing visibility and value!ings, largely in isolation, but it's good that lots of experimentation is going on

# data[].attributes.headline
S2R2 Questions to ask your supervisor at your RSE performance review, or at an RSE interview AND Let's discuss employment realities, as a student, as a grad, as a postgrad, as an ECR ...

# data[].attributes.subject
S2R2 Questions to ask your supervisor at your RSE performance review, or at an RSE interview AND Let's discuss employment realities, as a student, as a grad, as a postgrad, as an ECR ...

# data[].attributes.body
(Annual) performance review is an extremely valuable activity, especially if it also involves planning your goals for the next review. It's an opportunity to sit back and think about your skills and development, in addition to your immediate actions/projects/outputs, and to get feedback from your supervisor or manager about how to best support your growth and to find strategic opportunities to collaborate and demonstrate your value.

# data[].attributes.headline
S2R2 Questions to ask your supervisor at your RSE performance review, or at an RSE interview AND Let's discuss employment realities, as a student, as a grad, as a postgrad, as an ECR ...

# data[].attributes.subject
S2R2 Questions to ask your supervisor at your RSE performance review, or at an RSE interview AND Let's discuss employment realities, as a student, as a grad, as a postgrad, as an ECR ...

# data[].attributes.body
What are the questions one should ask or just sharing experiences to set a benchmark for expectations for RSEs?

"As a student, I'm kind of curious what sort of jobs I should be looking for after graduation, or what kind of undergraduate opportunities are available." This is a great jumping off point for what could be a wide-ranging discussion on the realities and expectations of employment in the field.

# data[].attributes.headline
S2R1 What exactly does a senior RSE look like? Tales from UniMelb

# data[].attributes.subject
S2R1 What exactly does a senior RSE look like? Tales from UniMelb

# data[].attributes.body
There is teaching, managing & strategy, and then technical.

UniMelb has an academic specialist. Much better fit for Adrian.

Dougie Boyle is one of us!

Promotion has Achievement relative to Opportunity at UniMelb.

If they had to outsource this externally, it would cost a lot!!!

# data[].attributes.headline
S2R1 What exactly does a senior RSE look like? List of Professors and Associate Professors

# data[].attributes.subject
S2R1 What exactly does a senior RSE look like? List of Professors and Associate Professors

# data[].attributes.body
List of professors and a/prof:

Professor Neil Chue Hong - https://www.software.ac.uk/about/staff/person/neil-chue-hong

Professor Simon Hettrick https://society-rse.org/about/governance/simon-hettrick/

A/Prof Adrian Bickerstaffe
Principal Research Fellow- System Analyst - https://findanexpert.unimelb.edu.au/profile/342653-adrian-bickerstaffe

Person X - I will start a new position as professor/chair of software engineering

Prof Dougie Boyle -https://findanexpert.unimelb.edu.au/profile/142641-dougie-boyle

# data[].attributes.headline
What do you need to feel safe at this unconference?

# data[].attributes.subject
What do you need to feel safe at this unconference?

# data[].attributes.body
Anonymous

let me refactor that for you ``` LAUGH = False ```

Anonymous

Very true regarding jobs, not everyone has the luxury of attending professional events just to be social some people need to make rent

Anonymous

I feel very safe with this slido way of communicating.

Anonymous

Supportive, no judgment words

Anonymous

if(terrible_code == TRUE) { dont_laugh() } else { still_dont_laugh() }

Anonymous

Keep things PG-13

Anonymous

I feel safe in this format, like the way it is being run.

Anonymous

Don't be shy or judgemental about asking about jobs/opportunities

Anonymous

brave? this is brave ><)))>

Anonymous

Anonymous makes me more brave. ohahaha

Anonymous

say something crazy urself so we feel like its ok for us to do the same

Anonymous

Remind me to let go of ego

Anonymous

Don't refer to groups of people as "guys"

Anonymous

Anonymous and no pressure to join social events, I get anxious speaking to others on a personal level, I'm better chatting professionally

Anonymous

But seriously, anonymity is fine, and in raising issues, speakers can avoiding naming names or being too specific about the workplace, colleagues, etc.

Anonymous

Find our commonalities - sharing stories of our challenges and mistakes helps everyone else feel like they're not alone!

Anonymous

Anonymity for the win. The abyss is deep!

Anonymous

let's not call any question / answer "simple", "trivial", "obvious", "of course", etc

Anonymous

anonymity

Anonymous

The first rule of RSEAA is...

Anonymous

Allow some space to discuss problems with being an RSE

Anonymous

I'm safe.

Anonymous

anonymity

Anonymous

the welcoming environment already makes me feel safe!

Anonymous

Let me stay at Rowlands place and watch tv

Anonymous

Know what is/is not recorded

Anonymous

What happens at RSEAA22 stays at RSEAA22

Anonymous

Do not record

Anonymous

I feel safe

Anonymous

maintain this anonymity

Anonymous

The anonymity's helping.

# data[].attributes.headline
Why do you think RSE skills and roles are important?

# data[].attributes.subject
Why do you think RSE skills and roles are important?

# data[].attributes.body
Anonymous

Lotta pitfalls to fall into with ML/AI/CS as a scientist

Anonymous

Providing a chance to know more people

Anonymous

How many of you lot are research system administrators as opposed to research software developers?

Anonymous

Shaping & sharing best practices for data analysis / research software

Anonymous

Existing platforms in academia are not remotely adequate to accommodate their data needs both in processing, storage and security. ESPECIALLY SECURITY. Everyone needs their data protected, particularly if you are working with marginalised communities, (especially Indigenous groups) they definitely need their data protected!!!

Anonymous

Wow there is a lot of opinions on this one. Good!

Anonymous

Teamwork makes the dream work!

Anonymous

Not everyone can do everything and learn all skills, we need to work together with a variety of skillsets

Anonymous

Importance of other forms of bridge-building than technical - being able to cooperate with the enterprise IT departments, research integrity and RDM teams, etc

Anonymous

We can compare RSE with librarians. Why librarians are valuable? Old Egypt already new the librarian value, see Great Library of Alexandria.

Anonymous

85% of research already requires digital skills.

Anonymous

So much data, not enough tools do deal with it!

Anonymous

There is a lot of research component. I feel that I get to work on multiple smaller projects that I want. Many RSE can fail and it is still fine. Most commercial software are built on tombs of RSEs.

Anonymous

Software is a craft, it requires engineering effort. We are the effort.

Anonymous

Most researchers are not software engineers or computer scientists.

Anonymous

When doing science, it helps to know how to write good code. When writing scientific code, it helps to understand the science.

Anonymous

To make science more reproducible

Anonymous

Academics in developing countries can easily access open source research software, contributing to global equity!

Anonymous

Bringing good coding practices to research, but with an understanding o f the unique research cultures

Anonymous

And software needs research

Anonymous

Because otherwise you'd have to add another couple of years to every doctoral candidature.

Anonymous

There are underlying CS & engineering practices that cut across all research domains

Anonymous

It's the information age and computers underpin and drive heaps of contemporary research. Much research simply wouldnt happen without RSEs

Anonymous

Translation between domain knowledge and software engineering helps facilitate good decision making

Anonymous

bridge the skills from coding practise with science expertise to create better reproducible outputs for society

Anonymous

They are/will be essential for the future of research.

Anonymous

Research needs software.

Anonymous

There is a huge gap between research prototype software and more broadly useful software - and it's such a waste when research code never ends up used by others!

Anonymous

I'm still unsure what RSEs are

Anonymous

Help bring the great ideas of researchers to the world.

Anonymous

so that the research world does not crumble completely

Anonymous

Community converges on best practices that inform culture.

Anonymous

Connecting people from different backgrounds

Anonymous

Scientific software underlies a huge amount of research and research funding, so it had better be done well!

Anonymous

To facilitate multidisciplinary discussion and understanding

# data[].attributes.body
Anonymous

I reckon a lot of us are facing the same sorts of problems. Ongoing funding for our roles, job security, commensurate recognition for intellectual contributions, etc.

Anonymous

To meet a group of remarkable people ... to see if they could become something more ... so when we needed them ... they could fight the battles that we never could

Anonymous

To become a freelancer

Anonymous

To see how others are addressing the academia - software development "abyss"

Anonymous

To get a better idea about career prospects in RSE (as opposed to the traditional academic career path)

Anonymous

to complain to sympathetic ears

Anonymous

Me too

Anonymous

The team is a lie.

Anonymous

Ill join the team

Anonymous

looking to join the team!

Anonymous

Looking for people to join our team

Anonymous

I want to find out what RSE is about

Anonymous

Job plz

Anonymous

I am in for the devops paper

Anonymous

To get a better idea about career prospects in RSE (as opposed to the traditional academic career path)

Anonymous

want to get started with an RSE career!

Anonymous

Gather requirements for a Web Platform for RSEs

Anonymous

I've been in open source for 3 years but didn't know about RSE until this year...

Anonymous

I just want to find out what job opportunities are available in RSE, hoping someone will talk about available jobs?

Anonymous

Meet People!

Anonymous

Discuss ideas for a devops to resops paper

Anonymous

I've flittered in and out of RSE career and interesting to see what other people have been up to!

Anonymous

I heard rumours of an RSE community and this I had to see.

Anonymous

Want to move away from corporate SE permanently

Anonymous

Connect with other RSEs to help and get help, procrastinate from writing grants

Anonymous

I wanted to see what RSEs are doing and their struggles + experiences.

Anonymous

To meet the people!

Anonymous

Building community projects

Anonymous

passion for Research Software and to meet new people

Anonymous

Networking

Anonymous

Rowland said it would be cool

Anonymous

I would like to see RSE community progress, and meet community.

Anonymous

My lecturer invited me, and it sounded imformative and interesting....also it was paid for

Anonymous

Connection with the community

Anonymous

To connect with people in RSE domain

Anonymous

To meet other RSEs and find out how diverse a community we are

Anonymous

to know that's happening in the RSE domain

Anonymous

Community is about showing up

Anonymous

Saranjeet said it would be cool

Anonymous

Rowland emailed me repeatedly

Anonymous

Networking

Anonymous

Meet fellow nerds

Anonymous

Wanted to engage more with the RSE community, curious what people had to say!

Anonymous

To support the community

Anonymous

Explore the community

Anonymous

Rowland invited us. : )

Anonymous

Rowland said it would be cool

Anonymous

QCIF Let me know about it

Anonymous

Meet other RSEs and see who else is out there in an emerging and important role.

Anonymous

Community

Anonymous

Connect with RSEs

Anonymous

More networking

Anonymous

As a follow-on for the RSE AUNZ conference

Anonymous

Yes

Anonymous

I wanted to meet my tribe

Anonymous

To see what others in our field are up to and connect with the community

# data[].attributes.attachment_link.title
Results - can you speak more than one language

# data[].attributes.body
1. Tom shared the Turing Way four-way grid for reproducibility which sparked interesting chat about reproducibility being different for different users and use-cases

https://the-turing-way.netlify.app/reproducible-research/overview/overview-definitions.html#table-of-definitions-for-reproducibility

2. It was pointed out that reproducible environments can be difficult to achieve. Simply using a docker container was insufficient.

3. A final contribution was the importance of explaining what reproducibility is to scientists and users, as this is not widely or well understood.

# data[].attributes.attachment_link.readable_text
The most common definition of reproducibility (and replication) was first noted by Claerbout and Karrenbach in 1992 [[ CK92] and has been used in computational science literature since then. Another popular definition has been introduced in 2013 by the Association for Computing Machinery (ACM) IT18], which swapped the meaning of the terms 'reproducible' and 'replicable' compared to Claerbout and Karrenbach.

The following table contrasts both definitions [ HBP+18].

Term

Claerbout & Karrenbach

ACM

Reproducible

Authors provide all the necessary data and the computer codes to run the analysis again, re-creating the results.

(Different team, different experimental setup.) The measurement can be obtained with stated precision by a different team, a different measuring system, in a different location on multiple trials. For computational experiments, this means that an independent group can obtain the same result using artifacts which they develop completely independently.

Replicable

A study that arrives at the same scientific findings as another study, collecting new data (possibly with different methods) and completing new analyses.

(Different team, same experimental setup.) The measurement can be obtained with stated precision by a different team using the same measurement procedure, the same measuring system, under the same operating conditions, in the same or a different location on multiple trials. For computational experiments, this means that an independent group can obtain the same result using the author's artifacts.

Barba (2018) [ Bar18] conducted a detailed literature review on the usage of reproducible/replicable covering several disciplines. Most papers and disciplines use the terminology as defined by Claerbout and Karrenbach, whereas microbiology, immunology and computer science tend to follow the ACM use of reproducibility and replication. In political science and economics literature, both terms are used interchangeably.

In addition to these high level definitions of reproducibility, some authors provide more detailed distinctions. Victoria Stodden [ Sto14], a prominent scholar on this topic, has for example identified the following further distinctions:

Computational reproducibility: When detailed information is provided about code, software, hardware and implementation details.

Empirical reproducibility: When detailed information is provided about non-computational empirical scientific experiments and observations. In practice, this is enabled by making the data and details of how it was collected freely available.

Statistical reproducibility: When detailed information is provided, for example, about the choice of statistical tests, model parameters, and threshold values. This mostly relates to pre-registration of study design to prevent p-value hacking and other manipulations.

Table of Definitions for Reproducibility

At The Turing Way, we define reproducible research as work that can be independently recreated from the same data and the same code that the original team used. Reproducible is distinct from replicable, robust and generalisable as described in the figure below.

#

The different dimensions of reproducible research described in the matrix above have the following definitions:

Reproducible: A result is reproducible when the same analysis steps performed on the same dataset consistently produces the same answer.

Replicable: A result is replicable when the same analysis performed on different datasets produces qualitatively similar answers.

Robust: A result is robust when the same dataset is subjected to different analysis workflows to answer the same research question (for example one pipeline written in R and another written in Python) and a qualitatively similar or identical answer is produced. Robust results show that the work is not dependent on the specificities of the programming language chosen to perform the analysis.

Generalisable: Combining replicable and robust findings allow us to form generalisable results. Note that running an analysis on a different software implementation and with a different dataset does not provide generalised results. There will be many more steps to know how well the work applies to all the different aspects of the research question. Generalisation is an important step towards understanding that the result is not dependent on a particular dataset nor a particular version of the analysis pipeline.

More information on these definitions can be found in "Reproducibility vs. Replicability: A Brief History of a Confused Terminology" by Hans E. Plesser [ Ple18].

#

Reproducible But Not Open

The Turing Way recognises that some research will use sensitive data that cannot be shared and this handbook will provide guides on how your research can be reproducible without all parts necessarily being open.

# data[].attributes.attachment_link.description
The most common definition of reproducibility (and replication) was first noted by Claerbout and Karrenbach in 1992 [[ CK92] and has been used in computational science literature since then. Another popular definition has been introduced in 2013 by the Association for Computing Machinery (ACM) IT18], which swapped the meaning of the terms 'reproducible' and 'replicable' compared to Claerbout and Karrenbach.

# data[].attributes.body
We discussed this issue in general and more specifically also Data storage. The main issues we identified were:
- Persistence of data and SW beyond any researcher's affiliation with an institution
- Access to read and write large amounts of data (TB-> PB)
- Cost and Governance of data stores / sw repos
- Our responsibilities as researchers, esp as supervisors, having an awareness of disciplinary expectations and responsibilities
- ongoing maintenance of systems both for storage, and functionality-as-a-service.

Some interesting links:
- https://researchdata.edu.au/  (appears not to require an AAF affiliation)
- https://datacentral.org.au/
- https://figshare.com/
- https://docs.globus.org/
- https://ardc.edu.au/services/nectar-research-cloud/
- https://www.qcif.edu.au/training/course-catalogue/

# data[].attributes.attachment_link.description
Find, access, and re-use data for research - from over one hundred Australian research organisations, government agencies, and cultural institutions.

# data[].attributes.body
Is there still no valid quantification for reproducibility improvement work?

# data[].attributes.body
1. Tom Honeyman mentioned that integrating computational environment or platform may be a way to improve reproducibility as it is hard to re-run the code in different environments / platforms

2. Aaron Willcox raised the concern in reproducibility that researchers are reluctant to data sharing and code sharing

3. Aidan Heerdegen talked about the problem caused by lacking reproducibility that one python library using -ffast-math can cause other libraries to alter their floating point result.
https://moyix.blogspot.com/2022/09/someones-been-messing-with-my-subnormals.html

-- Peiyu Wu

# data[].attributes.headline
S1R2: Quantify RSE outputs: Newcomer Hours and Videos

# data[].attributes.subject
S1R2: Quantify RSE outputs: Newcomer Hours and Videos

# data[].attributes.body
Julian Giordani expressed interesting in holding a Newcomer Hour or video series, similar to NumPy Newcomers Hour, for his software. Invite people in process, behind-the-scenes of the codebase and humanize the developers. + quantified by view numbers

Example video:

Ufuncs and dtypes: New possibilities in NumPy :: Sebastian Berg

However, each project needs to access their bandwidth and funding to explain beyond immediate users. NumPy Newcomers Hour hosted by NumPy Contributor Experience Lead, Inessa Pawson.

Contributor Experience Lead role funded by Chan Zuckerberg Imitative, so there is funding and recognition = possible!
-Mars Lee

# data[].attributes.body
Quantification of RSE output using workflow repositories (such as Dockstore)

# data[].attributes.body
1. Sayantika Banik has this cool project called https://github.com/sayantikabanik/DataJourney which is a tutorial for data engineering where you can do it without installing anything.

2. Nicholas Tan talked about Maurice Ling's work in Statistical Recipes in Python https://github.com/mauriceling/mauriceling.github.io/wiki/Statistical-Recipes-in-Python

3. Eslam Ibrahim shared this with us a project looking at comparing genomes https://github.com/Southern-Cross-Plant-Science/HEMA_ver1

4. Sayantika Banik suggested this https://evidentlyai.com/ after Rowland talked about https://www.canva.com/design/DAE7wJ12BRg/D5SAFKnYY1EUjnJU7RDBNg/view?utm_content=DAE7wJ12BRg&utm_campaign=designshare&utm_medium=link&utm_source=homepage_design_menu#1

# data[].attributes.attachment_link.title
GitHub - sayantikabanik/DataJourney: Work in progress effort, featuring data engineering/analytics workflows developed with open source tools ✨

# data[].attributes.attachment_link.description
Tutorial featuring Data engineering workflow and Open Source tools and technologies.

# data[].attributes.headline
Quantify RSE outputs: Scientific Communication with Comics

# data[].attributes.subject
Quantify RSE outputs: Scientific Communication with Comics

# data[].attributes.body
Comics as high level and engaging summary of RSE output. Similar to a newsletter.
Example: Comics for "Write Alt Text for Scientific Diagrams and Documentation", https://heyzine.com/flip-book/f3c7f85cdc.html
Inspiration for Execution: Applied Comics, https://appliedcomicsetc.com/projects/collaborations/
-Mars Lee

# data[].attributes.attachment_link.description
How To Write Alt Text for Scientific Diagrams and Documentation. By Mars Lee.. PDF to flipbook converted using heyzine.com

# data[].attributes.headline
Quantify RSE outputs: Depends on target audience and platforms

# data[].attributes.subject
Quantify RSE outputs: Depends on target audience and platforms

# data[].attributes.body
Academia->Academia = Citation
Academia without formal paper->Academia = Zenodo, DOI
Academia->Github = CITATION.cff, https://citation-file-format.github.io/
Academia->Emails = Newsletter
Academia-> General public, not academia or developer = ... comics?

Scientific Communication: Meet where people are already at and speak their language
-Mars Lee

# data[].attributes.body
Is the amount of time RSE has spent making its software better a fair measurement?

# data[].attributes.body
GitHub statistics?
https://citation-file-format.github.io/ ?