Question

People in a laboratory, working one the same project, generate all kind of files (fasta, images, raw data, statistics, readme.txt, etc...) that will be moved in some directories. How do you manage the hierarchy of those directories ?

• there is no standard hierarchy and the file are dropped anywhere. It all relays on the common knowledge.
• there is a clearly defined hierarchy (PROJECT_NAME/DATE/machine_user_image_result_but_this_is_the_second_run_because_1st_failed.txt...)
• files are uploaded on a wiki (you wouldn't do that for large files)
• there is a central file/wiki answering what/where is a file
• there is a Readme.txt/describe.xml in each folder.
• there is a tool (?) for managing this kind of information ?
• (...) ?

Thanks

Pierre

Answer 1

In my local computer, I have:

• a 'workspace' folder, in which each sub-folder correspond to a separate project
• a 'data' folder where I put all the data used by more than a project
• an 'archive' folder with all finished project

Within each project folder, I have:

• bugs/ -> I used to use ditz to keep track of bugs and To-Dos, but now I use only A7 hand-written papers
• data/
  • folders containing the different data I need to use, soft-linked from ~/data
• parameters/ -> ideally, I should have configuration files so if I want to run my analysis on other dataset, I only have to change the parameters here
• src/ -> with all code
  • a Makefile to re-run all the analysis I wish
  • scripts/ with all the scripts
  • lib/ eventually, if I am reusing code from other projects
  • pipelines/ with all .mk (makefile) files
• results/
  • tables/ -> tabular-like results
  • plots/ -> plots
  • manuscript/ -> draft for the manuscript, final figures and data, etc..
    • figures/
    • tables/
    • references/

I use git for revision control, to get a log of all the changes I make to scripts and results. Lately I have been reading about sumatra, and planning to give it a try (a slideshow for the curious here)

I still have to decide very well where to put .Rdata files, as I am still a novice to R.

note: you probably know already the article by Plos "A Quick Guide to Organizing Computational Biology Projects"

Answer 2

• For analyzed results, documentation, presentation (pdf, ppt, doc, xls) we are using eRoom provided by provided by EMC Corporation
• For experimental results it is a mix of your bullet 1 and 2 : There is a clearly defined hierarchy but after a while it all relays on the common knowledge to retrieve information when you want it in urgence.
• Some groups are experimenting ELN providing by CambridgeSoft
• We are also trying to create small social databases (ie an antibody database where people can share / retrieve their Western Blot experiments in order to avoid different people to test the same antibodies - Theye are able to "rate" the antibodies tested.)

Answer 3

Great question, thanks!

In my opinion there are several layers of files, and different approaches should be applied on each level. Here how it's organized in our lab.

1. Raw data (microarrays, for example)
   • Files are named and stored in clearly defined schema
   • Regular backup is mandatory
   • Some files, which we probably will not use further (like Affymetrix DAT files) are archived.
   • Access to the files is controlled
   • General information on experiments is stored in LIMS (we are using Labmatrix, but it's commercial)
   • We also store some preprocessed data (normalization, for example) if the procedure is clearly defined as SOP.
2. Temporary data (ongoing analysis)
   • Basically everybody are on their-own here. The files are usually stored locally and everyone responsible for their backup. I can access the data I need remotely (from home, for example).
   • I do keep some hierarchy based on projects, data type and analysis, but it's not strict and project-dependent.
   • I found Total Commander to be very useful for files management. For example, I can write a small comments for every file (Ctrl-Z), it's stored in a text file, and if I copy or move some file, the description goes with it.
   • Files to share for project team we keep on network shared drive with regular backup.
3. Results to share (documents, figures, tables, ...)
   • We are using Backpack from 37signals. Like wiki, but a little easier for non-tech users. Together with Basecamp for project management it's quite good, however it's again commercial and may not suit everybody.

Answer 4

one related tip is a handy bash script I got from:

http://dieter.plaetinck.be/per_directory_bash_history

which produces directory-specific bash histories (instead of one giant global history)

whenever I enter a directory I can easily access everything I ever did there, which is priceless when I am trying to remember what I actually did

Answer 5

I have following major directories:

/work - this is where I keep my project directories

/data - raw, unprocessed data

/software - 3rd party software required for the various work flows

/code - general code repo

I update the individual work directory as follows

/work
   | 
   /work/project1 
          |
          create sub-directory based on analysis 

For example code, analysis, results etc. Depending up on the repetitive nature of analysis, I create date based directories to track files generated at different time points. I also keep a README files with directories in project file to make it easier to check the content at a later stage. Big fan of "tree" whenever I need to check the contents in a directories. Irrespective of various data categories I deal with, this format worked for me.